Technical  Report  1229 


Training  to  Operate  a  Simulated  Micro-Unmanned 
Aerial  Vehicle  with  Continuous  or  Discrete  Manual 
Control 


Paula  J.  Durlach 

U.  S.  Army  Research  Institute 

John  L.  Neumann 

Dell  Corporation 

Deborah  R.  Billings 

University  of  Central  Florida 
Consortium  Research  Fellows  Program 


May  2008 


United  States  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences 


Approved  for  public  release;  distribution  is  unlimited. 


2008061 7  225 


U.S.  Army  Research  Institute 

for  the  Behavioral  and  Social  Sciences 

A  Directorate  of  the  Department  of  the  Army 
Deputy  Chief  of  Staff,  G1 

Authorized  and  approved  for  distribution: 

i>66zdJtJ 

BARBARA  A.  BLACK,  Ph.D.  MICHELLE  SAMS,  Ph.D. 

Research  Program  Manager  Director 

Training  and  Leader  Development 

Technical  review  by 

Jessie  Chen,  U.S.  Army  Research  Laboratory 
Scott  Beal,  U.S.  Army  Research  Institute 


NOTICES 

DISTRIBUTION:  Primary  distribution  of  this  Technical  Report  has  been  made  by  ARI. 
Please  address  correspondence  concerning  distribution  of  reports  to:  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences,  Attn:  DAPC-ARI-ZXM, 

2511  Jefferson  Davis  Highway,  Arlington,  Virginia  22202-3926. 

FINAL  DISPOSITION:  This  Technical  Report  may  be  destroyed  when  it  is  no  longer 
needed.  Please  do  not  return  it  to  the  U.S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences. 

NOTE:  The  findings  in  this  Technical  Report  are  not  to  be  construed  as  an  official 
Department  of  the  Army  position,  unless  so  designated  by  other  authorized  documents. 


REPORT  DOCUMENTATION  PAGE 


1.  REPORT  DATE 

May  2008 


2.  REPORT  TYPE 

Final 


3.  DATES  COVERED  (from. . .  to) 

January  2006  -  June  2007 


4.  TITLE  AND  SUBTITLE 

Training  to  Operate  a  Simulated  Micro-Unmanned  Aerial  Vehicle 
with  Continuous  or  Discrete  Manual  Control. 


5a.  CONTRACT  OR  GRANT  NUMBER 


5b.  PROGRAM  ELEMENT  NUMBER 

622785 


6.  AUTHOR(S) 

Paula  J.  Durlach  (U.S.  Army  Research  Institute); 
John  L.  Neumann  (Dell  Corporation); 

Deborah  R.  Billings  (University  of  Central  Florida) 


5c.  PROJECT  NUMBER 

A790 


5d.  TASK  NUMBER 

273 


5e.  WORK  UNIT  NUMBER 

H01 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences 

ATTN:DAPE-ARI-IF 

12350  Research  Parkway,  Orlando,  FL,  32826-3276 


8.  PERFORMING  ORGANIZATION  REPORT  NUMBER 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences 

251 1  Jefferson  Davis  Highway 
Arlington,  VA  22202-3926 


10.  MONITOR  ACRONYM 

ARI 


11.  MONITOR  REPORT  NUMBER 

Technical  Report  1229 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  is  unlimited. 


13.  SUPPLEMENTARY  NOTES 

Subject  Matter  POC:  Paula  J.  Durlach 


14.  ABSTRACT  ( Maximum  200  words): 

This  report  investigates  the  effects  of  continuous  vs.  discrete  control  methods  and  the  number  of  simultaneous 
camera  views  on  operator  performance  during  training  to  manually  control  a  simulated  micro-unmanned  aerial 
vehicle  (MAV).  Seventy-two  participants  were  trained  to  operate  a  MAV  in  a  simulated  environment,  to  designated 
criterion  levels.  They  were  then  given  training  missions  during  which  performance  was  measured.  Eight  conditions 
were  investigated,  formed  by  crossing  three  2-level  factors:  input  device  (mouse  vs.  game  controller),  input  control 
display  (discrete  vs.  continuous),  and  number  of  simultaneous  camera  views  (one  vs.  two).  Superior  performance 
was  observed  when  a  continuous  input  method  (e.g.,  multiple  degrees  of  freedom)  was  provided  for  continuous 
MAV  functions  (e  g.  maneuvering  in  space)  and  a  discrete  input  method  (e.g.,  single  action)  was  provided  for 
discrete  MAV  functions  (e.g.,  command  to  hover).  Under  these  conditions,  mission  times  were  shorter,  collisions 
were  fewer,  and  more  targets  were  photographed.  Effects  of  video  game  experience  and  spatial  ability  were  also 
investigated.  Recommendations  for  the  design  of  unmanned  vehicle  controls  were  discussed. 


15.  SUBJECT  TERMS  - - " - 

Spatial  ability,  video  game,  user-interface,  remote  control,  robot 

SECURITY  CLASSIFICATION  OF 

19.  LIMITATION  OF 

ABSTRACT 

Unlimited 

20.  NUMBER 

OF  PAGES 

67 

21.  RESPONSIBLE  PERSON 

Ellen  Kinzer 

Technical  Publications  Specialist 
703-602-8047 

16.  REPORT 

Unclassified 

17.  ABSTRACT 

Unclassified 

18.  THIS  PAGE 

Unclassified 

1 


ii 


Technical  Report  1229 


Training  to  Operate  a  Simulated  Micro-Unmanned 
Aerial  Vehicle  with  Continuous  or  Discrete  Manual 

Control 


Paula  J.  Durlach 

U.  S.  Army  Research  Institute 

John  L.  Neumann 

Dell  Corporation 

Deborah  R.  Billings 

University  of  Central  Florida 
Consortium  Research  Fellows  Program 


ARI-Orlando  Research  Unit 
Technology-based  Training 
Stephen  L.  Goldberg,  Chief 


U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
2511  Jefferson  Davis  highway,  Arlington,  Virginia  22202-3926 


May  2008 


Army  Project  Number  Personnel  Performance 

622785A790  and  Training  Technology 

Approved  for  public  release;  distribution  is  unlimited. 


in 


Acknowledgement 


The  authors  wish  to  acknowledge  the  Institute  for  Simulation  and  Training,  which  developed  the 
simulation  environment  and  provided  technical  support  under  a  contract  with  the  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI).  In  particular  we  wish  to  thank 
Glenn  Martin,  Duvan  Cope,  and  Casey  Thurston.  A  conference  proceedings  paper  was  published 
on  a  partial  data  set  from  the  experiment  (Neumann  &  Durlach,  2006). 


tv 


TRAINING  TO  OPERATE  A  SIMULATED  MICRO-UNMANNED  AERIAL  VEHICLE 
WITH  CONTINUOUS  OR  DISCRETE  MANUAL  CONTROL 

EXECUTIVE  SUMMARY 


Research  Requirement: 

Unmanned  systems  are  envisioned  to  be  a  key  part  of  the  Army’s  future  force.  At  the 
platoon  level.  Soldiers  will  have  the  Class  I  unmanned  aviation  system,  as  well  as  unmanned 
ground  sensor  systems.  The  Class  I  will  support  reconnaissance,  surveillance,  and  target 
acquisition.  Able  to  take-off  and  land  vertically  like  a  helicopter,  the  system  will  be  small 
enough  to  be  carried  in  the  field  by  dismounted  infantry  Soldiers.  Ideally,  the  Class  I  will  be 
relatively  easy  to  operate,  enabling  rapid  training  and  dispensing  with  the  need  for  specialized, 
dedicated  operators  (Office  of  the  Secretary  of  Defense,  2002). 

The  need  to  quickly  and  reliably  train  Class  I  operators  requires  that  the  human-system 
interface  be  intuitive  and  easy  to  master.  Moreover,  structured  training  must  be  developed  to 
ensure  that  training  is  efficient  and  effective.  Despite  this,  little  attention  has  been  devoted  to 
user  interface  design  or  training  development.  The  purpose  of  this  research  was  to  redress  some 
of  these  gaps  by  examining  the  effect  of  user  interface  design  during  training  to  pilot  a  simulated 
micro-unmanned  aerial  vehicle  (MAV)  in  manual  mode. 

Procedure: 

Seventy-two  participants  were  trained  to  operate  a  MAV  in  a  simulated  environment. 
Initial  training  involved  familiarization  with  the  user  interface  and  practice  exercises.  During  the 
practice  exercises,  operators  had  to  conduct  a  sequence  of  written  or  oral  instructions  within  a  set 
period  of  time.  They  then  had  to  navigate  the  MAV  around  two  different  flight  skill  tracks. 

They  were  given  five  attempts  to  complete  these  tracks  within  a  designated  amount  of  time  and 
without  any  collisions.  Operators  were  then  given  a  series  of  training  missions  during  which 
their  performance  was  measured.  Repetition  of  the  flight  skill  courses  allowed  for  an  assessment 
of  performance  improvement  during  the  course  of  training;  during  these  repetitions  performance 
measures  included  time-to-complete,  number  of  collisions,  and  subjective  workload.  There  were 
also  two  more  tactical  missions  in  which  target  detection  was  required.  During  these  missions 
measures  included  number  of  targets  detected,  number  of  collisions,  and  subjective  workload. 

Each  operator  participated  in  one  of  eight  conditions,  with  the  conditions  determined  by 
the  user  interface  configuration.  The  eight  conditions  were  formed  by  crossing  three  2-level 
factors:  input  device  (mouse  vs.  game  controller),  input  control  display  (discrete  vs.  continuous), 
and  number  of  simultaneous  camera  views  (one  vs.  two).  Regarding  the  input  device,  a  mouse 
was  used  as  a  surrogate  for  touch  screen  technology.  Many  operator  control  unit  (OCUs)  utilize 
touch-screens;  however  touch  inputs  register  less  reliably  than  mouse  inputs.  Consequently,  we 
used  a  mouse  to  better  equate  input  reliability  across  input  devices.  The  alternate  input  device 
was  a  game  controller.  Game  controllers  are  used  to  control  movement  of  virtual  characters  in 
virtual  environments,  analogous  to  how  a  MAV  operator  must  control  the  movement  of  the 
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MAV  using  sensor  imagery  from  the  MAV’s  cameras.  It  therefore,  seemed  like  a  natural  input 
device  for  this  task.  The  mouse  (or  any  point-and-click  method)  represents  a  discrete  method  on 
input,  in  that  one  and  only  one  command  can  be  given  at  a  time.  The  game  controller  represents 
a  continuous  method  of  input,  in  that  continuous  adjustments  of  the  thumb  sticks  on  the  game 
controller  (see  Figure  3)  translate  into  continuous  adjustments  in  MAV  flight.  The  second  factor 
manipulated  (input  control  display)  involved  how  the  input  controls  were  displayed  on  OCU 
screen  (see  Figures  1  and  2).  In  one  version,  the  display  was  discrete  in  that  specific  symbols 
represented  specific  maneuvers  (e.g.,  an  up-pointing  arrow  represented  upward  movement,  and  a 
down-pointing  arrow  represented  downward  movement).  In  the  alternate  version,  the  display 
was  continuous  in  that  display  elements  could  be  manipulated  along  a  continuum  (e.g.,  the 
location  of  a  dot  relative  to  a  home  position  could  be  moved  up  or  down  to  control  altitude).  The 
third  factor  was  number  of  simultaneous  camera  views.  Typical  OCUs  only  display  sensor 
imagery  from  one  camera  at  a  time.  In  this  study  we  examined  whether  providing  both  camera 
views  at  the  same  time  would  provide  any  advantage  in  maneuvering  the  MAV  and  avoiding 
collisions. 

Findings: 

Performance  improved  over  the  repetitions  of  the  flight  skills  tracks,  suggesting  that 
participants  had  learned  something  during  training.  It  was  not  possible  to  distinguish  the  relative 
contributions  of  skill  mastery  vs.  spatial  learning  to  this  improvement  because  all  missions  were 
run  in  the  same  simulated  environment.  Video  game  experience  and  spatial  ability  correlated 
with  measures  of  performance. 

Performance  was  better  when  there  was  a  better  match  between  the  continuity  of  the 
control  method  and  the  continuity  of  the  function  to  be  controlled.  When  maneuvering  the  MAV 
in  3-D  space,  participants  did  better  with  the  continuous  methods.  Mission  times  were  shorter, 
collisions  were  fewer,  and  more  targets  were  photographed.  For  example,  for  participants  using 
the  mouse,  the  continuous  control  interface  supported  faster  mission  completion,  compared  with 
the  discrete  control  interface.  In  the  former  case,  participants  were  able  to  control  maneuver  in 
different  dimensions  simultaneously,  whereas  in  the  latter,  maneuvers  in  only  one  dimension 
could  be  input  at  a  time.  In  contrast,  when  it  came  to  taking  pictures,  participants  using  the 
mouse  performed  better  with  the  discrete  interface  than  participants  using  the  continuous 
interface.  It  was  inferred  that  this  was  because  the  command  to  hover  could  be  issued  with  a 
single  input  in  the  former  (but  not  the  latter)  case.  Overall,  participants  using  the  game  controller 
performed  at  least  as  well  or  better  than  participants  using  the  mouse.  This  was  the  case  whether 
participants  rated  themselves  as  video-game  devotees  or  not.  The  game  controller  afforded  both 
continuous  control  for  continuous  functions  and  discrete  control  for  discrete  functions. 

Providing  both  camera  views  simultaneously  compared  with  only  one  camera  view  at  a 
time  did  not  aid  performance.  If  anything,  performance  was  poorer  when  both  views  were 
displayed  simultaneously.  This  may  have  been  due  to  the  reduced  size  of  the  sensor  windows  in 
the  dual  display  compared  with  the  single  display. 
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Utilization  and  Dissemination  of  Findings: 

The  training  protocols  and  performance  measures  developed  for  this  research  can  be 
applied  to  develop  simulation-based  structured  training  for  MAV  operator  training.  Further  work 
would  be  required  to  validate  whether  simulation  training  with  these  protocols  transfers  to  live 
mission  performance  with  actual  MAVs.  The  successful  use  of  flight  simulators  for  initial  and 
refresher  pilot  training  for  manned  vehicles  suggest  the  same  strategy  could  be  applied  for 
unmanned  vehicles.  The  training  missions  developed  can  also  be  used  as  research  protocols  to 
test  the  effects  of  variations  in  user  interface  design  on  operator  performance  and  acceptance.  SA 
Technologies  adopted  our  simulation  and  mission  protocols  in  order  to  investigate  the  effect  of 
operator  control  unit  screen  size  on  operator  performance. 

The  results  of  the  interface  manipulations  investigated  here  suggest  a  simple  heuristic  for 
designing  interface  control  methods.  The  control  method  should  match  the  function  of  the 
command.  A  discrete  command  (like  hover)  is  likely  best  implemented  as  a  single  discrete 
input.  Inputs  that  control  continuous  movements  through  space,  on  the  other  hand,  are  likely  best 
implemented  as  continuous  inputs.  Further  research  will  be  required  to  confirm  if  this  rule  of 
thumb  holds  up  under  a  number  of  use  cases. 
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TRAINING  TO  OPERATE  A  SIMULATED  MICRO-UNMANNED  AERIAL  VEHICLE 
WITH  CONTINUOUS  OR  DISCRETE  MANUAL  CONTROL 


Introduction 

The  U.S.  military  is  seeking  the  development  of  increasingly  smaller  and  lighter 
unmanned  aviation  systems  (UASs).  The  U.S.  Defense  Advanced  Research  Projects  Agency 
(DARPA)  began  supporting  the  development  of  one-half-kg  to  eight-kg  micro-aerial  vehicles 
(MAVs)  in  1997  and  is  currently  supporting  the  development  of  10-g  nano-aerial  vehicles 
(NAVs),  according  to  Flight  International  (2005).  The  motivation  for  small  UASs  is  to  provide 
reconnaissance,  surveillance,  and  target  acquisition  (RSTA)  at  the  small  unit  level  (e.g.,  platoons 
and  squads).  The  small  vehicle  size  is  intended  to  permit  transport  by  dismounted  Soldiers  and  to 
allow  maneuvering  in  tight  spaces  such  as  urban  canyons.  In  addition,  the  low  altitude  flight 
envisioned  for  these  systems  could  alleviate  the  need  for  air  space  coordination,  allowing  these 
small  UASs  to  be  launched  at  a  moment’s  notice.  Finally,  the  intention  is  that  these  vehicles  be 
easy  to  operate,  enabling  rapid  training  and  dispensing  with  the  need  for  specialized,  dedicated 
operators  (Office  of  the  Secretary  of  Defense,  2002). 

The  need  to  quickly  and  reliably  train  UAS  operators  requires  that  the  human-system 
interface  be  intuitive  and  easy  to  master  and  that  structured  training  be  developed;  however, 
developers  of  MAVs  and  NAVs  are  mainly  concerned  with  vehicle  technology.  Attention  to 
training  requirements  and  human-system  interaction  issues  are  often  neglected  until  late  in  the 
development  stage.  This  can  create  problems  such  as  the  fielding  of  equipment  without  clear 
standards  for  training  certification,  and/or  interface  design  features  that  are  not  well  suited  to  the 
needs  of  the  user. 

The  purpose  of  the  present  research  was  to  establish  how  variation  in  operator  interface 
design  would  impact  performance  during  training  to  manually  control  a  simulated  MA  V.  The 
report  also  provides  a  review  of  some  of  the  relevant  issues  involved  in  UAS  operator  training 
and  performance. 


Background 

In  manual  control  mode,  a  UAS  operator  can  control  maneuvers  of  the  vehicle  in  near- 
real  time  (as  opposed  to  way-point  pre-programmed  navigation,  which  is  determined  prior  to 
launch);  however,  unmanned  vehicle  system  design  may  constrain  the  operator’s  freedom  to 
maneuver.  Sometimes  these  constraints  are  imposed  to  lessen  the  training  burden.  For  example, 
Honeywell’s  2006  prototype  MAV  system  limited  the  velocity  of  the  MAV  to  six  knots  in 
manual  mode,  despite  the  capability  of  the  system  to  fly  at  a  speed  of  over  25  knots.  This 
constraint  was  intended  to  prevent  aerodynamic  instabilities,  which  could  occur  if  particular 
patterns  of  maneuver  commands  were  issued  when  the  MAV  was  traveling  at  faster  speeds.  By 
limiting  velocity  in  manual  mode,  the  need  to  train  operators  on  the  types  of  maneuvers  they 
should  not  attempt  at  higher  speeds  was  eliminated. 


Discrete  and  Continuous  Control  Schemes 


Even  though  system  constraints  can  be  beneficial  at  times,  at  other  times  these 
restrictions  may  not  optimally  support  operator  training  and  performance.  One  way  in  which 
UAV  system  design  may  potentially  constrain  users  involves  the  chosen  input  method  (discrete 
or  continuous  interfaces).  For  example,  discrete  input  interfaces  instruct  the  vehicle  to  make 
maneuvers  with  specific  predetermined  parameters  and  may  simplify  the  task  for  the  operator  by 
limiting  the  possible  inputs.  In  contrast,  continuous  input  interfaces  allow  the  user  to  make  fine¬ 
grained  adjustments  along  a  continuum  and  to  alter  more  than  one  parameter  at  a  time,  which 
may  help  operators  navigate  difficult  terrain  because  they  have  more  precise  control  of  the 
vehicle.  Figure  1  illustrates  a  discrete  input  control  display  (ICD).  By  selecting  one  of  the  arrows 
in  the  display  scheme,  the  operator  can  instruct  the  vehicle  to  move  up,  down,  laterally,  rotate,  or 
stop  (hover),  and  these  maneuvers  are  carried  out  at  a  predetermined  speed.  The  vehicle  keeps 
traveling  in  a  certain  direction  until  another  command  is  entered.  Selection  of  one  input 
command  cancels  the  previous  command;  thus  commands  can  be  given  for  only  one  degree  of 
freedom  of  movement  at  a  time  (though  inertia  can  result  in  maneuvers  at  oblique  angles  or  at 
various  speeds). 


Figure  1 .  Illustration  of  a  discrete  input  control  display. 

Figure  2  illustrates  a  more  continuous  input  control  scheme.  Here,  dragging  one  of  the 
dots  away  from  its  initial  “home”  location  causes  the  desired  maneuver  in  that  respective 
direction,  with  velocity  increasing  the  further  from  home  the  dot  is  dragged  (up  to  a  maximum  at 
the  edges).  The  top  arc  acts  like  a  steering  wheel,  controlling  rotation.  The  side  bar  controls 
altitude  (up-down),  and  the  middle  circle  controls  both  forward-backward  and  lateral  movement. 
A  command  to  fly  laterally  and  forward  simultaneously  can  be  given  with  one  drag  of  the  dot.  In 
addition,  in  this  continuous  input  scheme,  issuing  a  command  in  one  dimension  does  not  cancel 
previously  issued  commands  in  the  other  dimensions.  Thus,  the  operator  has  the  ability  to  control 
maneuvers  in  multiple  dimensions  (multiple  degrees  of  freedom)  at  one  time. 
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Figure  2.  Illustration  of  a  continuous  input  control  display. 

Existing  operator  interface  designs  for  prototype  and  operational  MAV  systems  have 
used  either  discrete  or  continuous  control  schemes,  or  some  blending  of  the  two;  however,  there 
has  been  little  research  on  how  these  design  decisions  affect  usability,  mastery,  or  workload. 
Strommen  (1993)  found  that  young  children  playing  a  “capture  the  number”  game  captured  more 
targets  with  a  discrete  control  scheme  than  with  a  continuous  control  scheme  if  they  were 
inexperienced  with  similar  game  environments;  however,  the  control  scheme  failed  to  affect 
target  capture  in  experienced  players.  It  was  suggested  that  discrete  control  may  be  especially 
helpful  for  novice  users  by  reducing  demands  on  working  memory.  Chen  and  Durfee  (1993) 
found  faster  target  acquisition  times  with  discrete  control  than  with  continuous  control  when 
adults  were  tasked  to  manipulate  a  simulated  robotic  arm.  The  input  methods  were  very  different, 
however;  discrete  commands  were  given  by  voice  (e.g.,  up,  down,  left,  right),  whereas 
continuous  commands  were  given  by  the  position  of  the  shoulder.  It  is  not  clear  whether  discrete 
shoulder  movements  would  have  been  superior  to  continuous  shoulder  movements. 

Input  Device  Issues 

The  example  of  the  robotic  arm  illustrates  how  the  method  of  operator  input  (i.e.,  voice 
vs.  shoulder  position)  could  be  a  factor  independent  of  or  interactive  with  input  scheme  (i.e., 
discrete  vs.  continuous).  Some  methods  of  command  input  fit  more  naturally  with  discrete 
commands  (e.g.,  voice),  while  others  are  more  conducive  to  continuous  control  (e.g.,  a  steering 
wheel).  Wickens  (1992)  suggested  that  discrete  voice  commands  are  not  well  suited  for 
conveying  spatial  and/or  continuous  input  compared  with  discrete  manual  (hand-guided)  input. 
Input  devices  for  unmanned  systems  vary  widely,  including  small  computers  using  touch-based 
input  (with  a  stylus),  joysticks,  wheels,  natural  language,  and  visual  gestures  (Chen,  Haas, 
Pillalamarri,  &  Jacobson,  2006).  In  the  present  experiment,  we  compared  the  use  of  two  different 
input  devices.  We  chose  one  input  device  that  seemed  to  naturally  complement  a  discrete  control 
scheme  and  one  that  seemed  to  naturally  complement  a  continuous  control  scheme.  For  the 
former,  we  used  a  standard  computer  mouse.  We  chose  the  mouse  as  a  surrogate  for  a  touch¬ 
screen  and  stylus  (a  frequently  used  input  method  for  MAVs).  Both  mouse  and  stylus  use  the 
point  and  click  approach  that  directs  the  focus  of  attention  away  from  the  camera  imagery 
view(s);  however,  with  the  mouse,  inputs  are  more  reliably  registered.  In  our  previous  research 
with  a  touch  screen  and  stylus,  users  became  frustrated  by  touches  that  failed  to  register 
(Durlach,  Neumann,  &  Bowens,  2006).  We  did  not  want  input  reliability  to  affect  our  measures 
in  the  current  research,  so  we  used  a  mouse  instead.  Previous  research  indicates  little  difference 
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in  speed  between  a  stylus  and  a  mouse  (Card,  English  &  Burr,  1978;  MacKenzie,  Sellen  & 
Buxton,  1991). 

Half  of  the  participants  in  the  present  investigation  received  training  on  simulated  MAV 
control  using  the  mouse.  Of  these,  half  entered  inputs  using  the  discrete  input  scheme  illustrated 
in  Figure  1,  and  the  rest  used  the  continuous  input  scheme  illustrated  in  Figure  2.  Thus,  the  effect 
of  a  strictly  discrete  input  scheme  vs.  a  more  continuous  input  scheme  could  be  assessed,  holding 
input  device  constant. 

While  half  of  the  participants  used  a  mouse,  the  remainder  of  the  participants  received 
training  on  simulated  MAV  control  using  a  dual-thumbstick  game  controller,  as  illustrated  in 
Figure  3.  Millions  of  people  use  this  type  of  device  to  navigate  through  3-D  virtual 
environments  while  playing  video  games,  and  this  is  essentially  what  the  operator  of  an 
unmanned  system  has  to  do.  This  type  of  game  controller  affords  a  more  continuous  form  of 
input  than  either  of  the  described  point-and-click  schemes.  Each  thumb  stick  can  be  used  to 
control  two  dimensions  of  movement,  and  they  can  be  operated  in  parallel.  Not  only  can 
gradations  in  speed  be  controlled  by  how  far  the  thumbstick  is  moved  in  any  one  direction,  but 
also  four  dimensions  of  movement  (up-down,  forward-back,  lateral  movement  left-right,  and 
rotation)  can  be  manipulated  in  parallel.  For  research  participants  equipped  with  the  game 
controller,  visual  feedback  on  their  thumb  movements  was  supplied  on  the  display.  Half  the 
participants  equipped  with  the  game  controller  had  an  interface  display  like  the  one  shown  in 
Figure  1 .  When  the  game  controller  was  employed  to  maneuver  the  MAV,  the  arrows 
corresponding  to  the  commanded  maneuvers  were  illuminated.  The  rest  of  the  participants 
equipped  with  the  game  controller  had  an  interface  display  like  the  one  shown  in  Figure  2.  For 
these  participants,  when  the  game  controller  was  employed  to  maneuver  the  MAV,  the  dots 
corresponding  to  the  commanded  maneuvers  repositioned  themselves  to  represent  the  current 
inputs. 
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Figure  3.  Illustration  of  the  game  controller  with  functions  labeled.  Left-right/lateral  movements 
(left  thumbstick)  did  not  change  vehicle  heading. 
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In  summary,  there  were  four  combinations  of  input  device  and  display  control  scheme, 
which  represented  different  degrees  of  continuity  in  the  operator’s  ability  to  control  the  MAV. 
The  combination  of  arrows  and  the  mouse  was  the  least  continuous.  The  combination  of  dots  and 
the  mouse  was  somewhat  more  continuous,  in  that  it  allowed  gradations  in  speed  and 
simultaneous  control  of  forward-backward  and  lateral  movement.  The  combinations  using  the 
game  controller  were  even  more  continuous  because  all  inputs  could  be  given  in  parallel.  The 
only  difference  in  these  conditions  was  whether  the  graphical  visual  feedback  provided  speed 
information  (dots)  or  not  (arrows). 

It  could  be  argued  that  the  more  discrete  the  input  scheme,  the  easier  it  should  be  to  learn, 
because  it  involves  managing  fewer  degrees  of  freedom  at  a  time.  On  the  other  hand,  more 
continuous  methods  provide  more  control.  They  could  take  longer  to  master,  but  could  ultimately 
lead  to  more  adept  maneuvering.  Learning  to  maneuver  with  the  game  controller  may  be 
particularly  challenging  for  those  unfamiliar  with  the  device  because  they  have  to  leam  the 
mapping  of  thumb  movements  to  MAV  responses.  Any  attention  toward  their  hands  could 
detract  from  attention  to  the  display  (both  sensor  imagery  and  feedback  scheme).  The  present 
research  sought  to  examine  these  questions  by  comparing  performance  during  training  across  the 
four  conditions. 

Sensor  Imagery  Issues 

In  order  to  employ  an  unmanned  system  for  RSTA,  the  operator  needs  to  interpret  the 
sensory  imagery  returned  from  the  vehicle  sensors.  Interpretation  of  sensor  imagery  is  important 
for  navigation,  obstacle  avoidance,  and  target  detection.  Electro-optical  or  infrared  cameras 
providing  streaming  video  are  the  most  frequently  employed  sensors  on  MAVs;  however,  these 
sensors  provide  an  impoverished  view  of  the  environment  due  to  limited  resolution,  limited  field 
of  view,  and  vehicle  vibration.  When  navigating  oneself  through  space,  there  is  a  close  coupling 
of  visual  and  kinesthetic  perception  with  action  and  continuous  feedback  between  them.  When 
navigating  an  unmanned  system  through  space,  perception  and  action  are  less  tightly  coupled, 
and  this  may  interfere  with  judgment  of  scale,  depth,  and  velocity  (Chen,  Haas,  Pillalamarri,  & 
Jacobson,  2006;  Peruch  &  Mestre,  1999;  Woods,  Tittle,  Feil,  &  Roesler  2004). 

Different  MAVs  vary  in  whether  their  cameras  are  fixed,  gimbaled,  or  have  zoom 
capabilities,  but  they  all  tend  to  carry  two  cameras.  One  camera  faces  forward  and  the  other 
downward.  Another  common  aspect  is  that  they  display  only  one  camera  view  to  the  operator  at 
a  time  due  to  bandwidth  limitations.  Because  of  this,  operators  need  methods  to  switch  between 
camera  views  and  to  cognitively  integrate  past  and  current  views  (Goodrich  &  Olsen,  2003). 
Based  on  our  past  research  (Durlach,  Neumann,  &  Bowens,  2006),  we  hypothesized  that  display 
of  both  camera  views  simultaneously  might  facilitate  MAV  maneuver  and  situation  awareness  by 
essentially  widening  the  field  of  view  and  allowing  easier  viewpoint  switching.  On  the  other 
hand,  simultaneous  displays  might  cause  the  operator  to  divide  their  attention  between  the  views, 
which  in  turn  might  cause  them  to  miss  important  information  (Durlach,  2004).  In  addition, 
because  the  simultaneous  views  took  up  the  same  screen  area  as  the  single  view,  each  camera 
image  was  smaller  than  the  single  camera  image,  and  this  might  have  negative  affects  on  piloting 
and/or  target  detection  (e.g.,  Stelzer  &  Wickens,  2006;  Wickens,  2005).  As  we  were  operating  in 
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a  simulation  environment  and  were  not  limited  by  bandwidth  considerations,  we  were  able  to  test 
these  possibilities.  Appendix  B  illustrates  the  different  camera  configurations  on  the  operator 
control  unit. 

In  summary,  for  this  experiment  three  aspects  of  the  operator’s  control  interface  were 
manipulated  in  order  to  assess  the  impact  on  performance  and  workload  during  training.  The 
three  factors  each  had  two  levels  and  were  crossed,  resulting  in  eight  between-group  conditions 
(2  x  2  x  2).  The  factors  were  input  control  device  (mouse  vs.  game  controller),  input  control 
display/scheme  (continuous  vs.  discrete),  and  number  of  sensor  views  available  at  a  time  (one 
camera  view  vs.  two  camera  views). 

Description  of  the  Simulated  MA  V 

The  simulated  MAV  was  based  loosely  on  Honeywell’s  2005  t-MAV  prototype,  as 
illustrated  in  Figure  4  (Kennedy,  Williams,  Robertson,  Pettitt,  &  Swiecicki,  2006),  with  vertical 
lift  and  land  as  well  as  hover  capabilities.  It  had  two  fixed  cameras  facing  forward  and 
downward,  which  could  send  streaming  video  to  the  operator  via  the  display  on  the  OCU. 

Similar  to  the  actual  prototype,  the  airspeed  of  the  simulated  MAV  was  limited  to  six  knots. 

As  no  standard  simulation  training  conditions,  methods,  or  assessments  existed  for  this 
prototype,  we  devised  our  own  training  methodology,  which  might  be  useful  for  adoption  in  the 
future.  Trainees  were  first  instructed  on  the  MAV’s  capabilities  and  the  basic  functions  of  their 
designated  OCU  configuration.  They  were  then  given  multiple  practice  exercises  in  which  they 
had  to  complete  a  series  of  specific  maneuvers  within  a  set  amount  of  time.  They  then  had  to 
complete  two  different  maneuver-skill  courses.  Finally,  if  participants  met  certain  criterion 
levels  they  went  on  to  complete  more  demanding  missions,  during  which  their  performance  was 
measured  to  assess  the  impact  of  the  OCU  manipulations. 
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Assessment  of  Individual  Differences 

We  considered  the  possibility  that  trainees  already  familiar  with  game  controllers  might 
have  a  distinct  advantage  over  participants  not  already  familiar  with  them.  We,  therefore,  asked 
participants  about  their  video-game  experience.  This  allowed  us  to  take  gaming  experience  into 
account  when  analyzing  the  results.  We  also  assessed  spatial  ability.  There  is  evidence  that 
spatial  ability,  as  assessed  by  psychometric  tests,  has  moderate  predictive  validity  for  spatial 
knowledge  acquisition  in  virtual  environments  (Waller,  Knapp  &  Hunt,  2001),  and  all  the 
performance  missions  involved  some  degree  of  navigation.  In  addition,  the  impact  of  number  of 
camera  views  might  depend  on  how  well  participants  were  able  to  mentally  integrate  the  forward 
and  downward  views  to  form  a  cognitive  representation  of  space.  Based  on  arguments  for 
multiple  types  of  spatial  intelligence  (Gardner,  1999;  Stumpf  &  Eliot,  1999;  Thurstone,  1938), 
we  gave  four  different  tests  intended  to  measure  different  aspects  of  spatial  ability. 

We  did  not  initially  attempt  to  equate  gender  across  conditions;  however,  we  realized 
there  was  a  strong  relationship  between  video  game  experience  and  gender  among  our  sample 
when  we  analyzed  a  partial  data  set  (28  participants)  for  a  conference  presentation  (Neumann  & 
Durlach,  2006).  We  found  that  the  males  reported  higher  levels  of  video  game  experience  than 
females  in  our  sample.  Thereafter,  we  subsequently  assigned  trainees  to  conditions  in  order  to 
ensure  that  an  equal  number  of  males  and  females  contributed  to  each  of  the  main  effects.  In  this 
way,  gender  served  as  a  rough  proxy  for  equating  video  game  experience  across  cells. 

Method 


Participants 

Thirty-six  male  and  36  female  participants  from  the  University  of  Central  Florida  area 
completed  the  experiment  in  exchange  for  $20  or  course  credit.  Three  other  people  did  not  meet 
initial  training  criteria  and  were  excused  from  the  experiment.  All  participants  were  at  least  1 8 
years  old  and  had  normal  color  vision.  The  median  age  was  19  years.  Each  participant  completed 
an  informed  consent  form  that  explained  the  scope  of  the  research  prior  to  any  testing. 

Apparatus 

The  participant  interacted  with  two  lap-top  computers.  One  of  these  displayed  the  OCU 
and  controlled  the  simulation  software.  The  other  presented  the  NASA-TLX  workload  survey 
(Hart  &  Staveland,  1988).  The  participants  interacted  with  these  computers  while  seated  at  a 
desk.  Another  computer  (desktop)  ran  OneSAF  Testbed  (OTB  version  2.5),  which  allowed  the 
injection  of  targets  into  the  terrain  database.  The  terrain  database  was  modeled  after  an  Army 
urban  training  site  at  Fort  Polk,  Louisiana. 
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The  Simulated  MA  V 


The  characteristics  of  the  simulated  MAV  were  loosely  based  on  the  t-MAV  developed 
under  the  Defense  Analysis  Research  Project  Agency’s  MAV  Advanced  Technology 
Demonstration.  The  MAV  was  a  ducted-fan  vertical  lift  vehicle  which  could  hover,  rotate  in 
place,  and  travel  at  an  airspeed  of  up  to  6  knots  under  manual  control.  Based  on  available 
information  about  the  t-MAV  we  developed  a  simple  flight  model,  which  caused  the  vehicle  to 
tilt  forward  one  degree  for  every  knot  of  forward  speed  and  which  gave  it  some  inertial 
properties  (e.g.,  when  in  forward  movement,  it  took  time  to  actually  stop  and  assume  a  hover 
after  the  hover  command  was  issued).  The  simulated  MAV  was  equipped  with  two  fixed 
cameras,  one  facing  forward,  and  one  facing  downward.  The  tilt  produced  by  forward 
movement  of  the  MAV  also  tilted  camera  angles  (e.g.  while  moving  forward,  the  downward 
camera  pointed  somewhat  behind  the  vehicle). 

Input  Devices 

Two  input  devices  were  used  to  operate  the  MAV.  One  was  a  standard  2-button/ 1 -wheel 
Dell  optical  mouse  with  a  USB  connector,  placed  on  a  mouse  pad,  and  the  other  was  a  Logitech 
dual-thumbstick  game  controller,  also  with  a  USB  connector  (see  Figure  3).  The  devices  enabled 
the  same  functions,  only  in  different  ways.  Mouse  clicks  on  OCU  display  activated  those 
controls,  whereas  selection  of  the  same  function  using  the  game  controller  activated  the  controls 
and  produced  the  same  visual  feedback  as  if  the  mouse  had  been  used. 

Operator  Control  Unit 

The  OCU  could  be  presented  in  one  of  four  configurations  (See  Appendix  B),  created  by 
crossing  two  two-level  factors:  camera  views  and  input  control  scheme  characteristics.  The  first 
factor  was  number  of  camera  views  displayed  at  a  time;  this  could  be  one  or  two  camera  views. 
When  there  was  only  one  overlapping  camera  view,  participants  could  switch  between  the 
forward  and  downward  camera  views  by  using  controls  displayed  on  the  OCU  tool  bar  (mouse 
condition)  or  buttons  on  the  game  controller.  Operators  could  take  still  photos  of  images  in  the 
active  view  by  using  controls  similar  to  the  camera  switching  controls.  When  there  were  two 
adjacent  camera  views,  the  active  camera  view  had  a  blue  border  at  the  top  of  the  view  frame  and 
a  target  indicator  in  the  center  of  the  image;  the  inactive  camera  view  had  a  grey  border  at  the  top 
of  the  view  frame.  Operators  could  take  still  photographs  of  images  in  the  active  camera 
window  and  could  choose  which  window  was  active  using  controls  displayed  on  the  OCU 
toolbar  (mouse)  or  buttons  on  the  game  controller.  All  participants  received  auditory  feedback 
when  they  took  a  snapshot  (sound  of  a  traditional  camera  shutter). 

The  second  factor  manipulated  was  the  ICD.  It  could  appear  in  one  of  two  versions, 
which  will  be  referred  to  as  continuous  or  discrete.  In  the  mouse  conditions,  participants  clicked 
on  the  ICD  to  control  the  movement  of  the  MAV.  In  the  continuous  case,  the  mouse  was  used  to 
drag  the  dots  to  produce  changes  in  rotation  (upper  arc),  altitude  (bar),  or  forward/backward  and 
lateral  (circle)  movement.  The  further  away  a  dot  was  dragged  from  the  center  of  the  circle, 
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the  faster  the  MAV  would  travel  (up  to  a  maximum  of  6  knots  at  the  edges).  Dots  could  be  left 
in  position  to  maintain  speed  and  heading  while  the  mouse  was  used  for  other  functions  (e.g., 
taking  a  snap  shot  or  manipulating  one  of  the  other  dots). 

In  the  discrete  case,  the  mouse  was  used  to  click  on  one  of  the  directional  arrows  or  a  tool 
bar  icon.  Upon  a  click,  the  icon  or  arrow  was  highlighted  to  indicate  it  was  selected.  Selecting 
one  of  the  arrows  activated  movement  in  the  selected  direction  and  cancelled  any  ongoing 
movement  in  another  direction  (with  some  inertia,  so  that  the  change  was  graded).  After 
selection,  movement  in  the  chosen  direction  increased  to  6  knots  and  then  stayed  at  that  speed 
until  another  maneuver  icon  was  selected.  Clicking  on  the  “X”  would  cause  all  movement  to 
slow  until  the  MAV  came  to  a  stable  hover. 

In  the  game  controller  conditions,  the  ICD  provided  the  same  visual  feedback  (dots 
moved,  or  arrows  and  icons  illuminated).  The  number  of  camera  views  and  ICD  were  crossed  to 
produce  the  four  different  OCU  display  configurations.  These  in  turn  were  crossed  with 
command  input  device  to  create  eight  conditions  in  total. 

Besides  these  aspects,  all  OCU  configurations  displayed  information  to  assist  mission 
execution.  An  altimeter  was  displayed  to  the  left  of  the  sensor  imagery  (or  camera  view). 
Airspeed  was  displayed  in  the  ICD.  A  satellite/map  view  of  the  terrain  was  presented  to  the  right 
of  the  sensor  imagery  (the  situational  awareness  view).  During  the  experiment,  the  map  scale  and 
center  were  fixed.  The  current  position  of  the  MAV  was  always  displayed  on  this  map. 

Timing  of  each  exercise  was  initiated  by  the  operator.  When  the  operator  activated  the 
take-off  icon  and  the  MAV  started  to  ascend,  the  timer  was  triggered.  Timing  ended  when  the 
MAV  was  grounded.  Mission  duration  was  displayed  in  the  upper  right  comer  of  the  OCU 
display.  Each  exercise  or  mission  commenced  with  a  launch  from  a  predetermined  spot  and 
ended  with  a  landing  at  a  point  designated  in  the  instructions.  Upon  launch,  the  MAV  rose  to  an 
altitude  of  60  feet  above  ground  and  maneuver  commands  issued  during  this  rise  had  no  effects. 
When  the  MAV  reached  take-off  altitude,  a  red  stop  sign  icon  on  the  menu  bar  illuminated.  At 
this  point,  maneuver  commands  could  be  given. 

Procedure 

Each  participant  completed  two  sessions  on  separate  days.  In  session  1 ,  which  lasted 
about  an  hour,  participants  completed  a  demographics  questionnaire  (see  Appendix  D)  and  four 
tests  of  spatial  ability.  The  demographic  survey  contained  questions  on  each  user’s  computer 
usage  and  video  game  playing  habits,  as  well  as  demographic  data  like  gender,  age,  and 
education.  The  spatial  ability  tests  included  the  Hidden  Figures  and  Hidden  Patterns  Tests  (ETS, 
1976),  and  the  Cube  Comparison  Test  (ETS,  1976),  as  well  as  a  spatial  orientation  test  developed 
after  Gugerty,  Brooks,  and  Treadway  (2005).  The  Hidden  Figures  and  Hidden  Patterns  Tests 
from  Educational  Testing  Services  (ETS)  measure  flexibility  of  closure,  also  known  as  field 
independence.  This  involves  picking  out  specific  patterns  or  shapes  embedded  within  distracting 
information.  The  Cube  Comparison  Test  from  ETS  examines  the  ability  to  rotate  images 
mentally  in  3-D.  The  spatial  orientation  test  measures  the  ability  to  judge  the  cardinal  direction 
from  a  building  to  a  target  from  different  viewing  directions. 


9 


MAV  operator  training  took  place  during  session  2,  which  lasted  two  to  three  hours.  For 
this  session,  participants  were  semi-randomly  assigned  to  one  of  eight  conditions  determined  by 
a  2  x  2  x  2  between-groups  design,  where  the  three  factors  were  input  device  (mouse  or  game 
controller),  ICD  (continuous  or  discrete),  and  number  of  camera  views  (one  or  two).  The 
number  of  males  and  females  in  each  condition  is  shown  in  Table  1 . 


Table  1 


Condition 

Males 

Females 

Device 

Camera  Views 

Display 

Mouse 

1 

Continuous 

4 

5 

Mouse 

1 

Discrete 

4 

5 

Mouse 

2 

Continuous 

4 

5 

Mouse 

2 

Discrete 

5 

4 

Game  Controller 

1 

Continuous 

4 

5 

Game  Controller 

1 

Discrete 

6 

3 

Game  Controller 

2 

Continuous 

4 

5 

Game  Controller 

2 

Discrete 

5 

4 

Each  operator  completed  the  training  equipped  with  a  printed  manual  (see  Appendix  C) 
and  facilitated  by  the  experimenter  (the  second  author).  Trainees  were  first  instructed  on  the 
MAV’s  capabilities  and  the  basic  functions  of  their  designated  OCU  configuration.  The 
experimenter  then  administered  and  graded  a  short  written  test  designed  to  assess  the  operator’s 
knowledge  of  all  the  commands  and  interface  components  required  for  the  research  (see 
Appendix  E).  Any  errors  were  re-trained.  Participants  were  then  given  seven  exercises  in  which 
they  had  to  complete  a  series  of  specific  maneuvers  within  a  set  amount  of  time.  For  the  first 
three  exercises,  participants  read  a  list  of  maneuvers  to  complete.  For  the  fourth  and  fifth 
exercises,  participants  followed  a  series  of  verbal  commands  read  aloud  by  the  experimenter.  As 
an  example,  Table  2  gives  the  instructions  participants  read  for  Practice  Exercise  2.  See 
Appendix  C  for  complete  participant  training  manual. 

Table  2 

Instructions  for  Practice^  Exercise^  2  _ 

Practice  Exercise  2: _ 

1 )  Press  OK  to  start  the  simulation  and  timer. 

2)  Execute  the  Take-off  command. 

3)  At  or  before  the  completion  of  take-off,  activate  the  view  window  for  Camera  1  (forward 
view). 

4)  Take  a  snapshot  with  camera  1 . 

5)  Activate  the  view  window  for  camera  2  (down  view). 

6)  Take  snapshot  with  camera  2. 

7)  Execute  the  Land  command. 

8)  This  must  be  completed  in  40  seconds  (:40)  or  less. _ 


While  the  first  five  exercises  involved  simple  maneuvers,  the  final  two  practice  exercises 
involved  negotiating  timed  courses.  Before  executing  each  exercise,  the  participant  was  allowed 
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to  observe  the  MAV  flying  the  course  in  automated  mode.  The  “racetrack”  course  required 
operators  to  pilot  the  MAV  for  one  lap  around  an  oval-shaped  course  running  through  a  village 
and  then  land  on  top  of  a  marked  rooftop.  Red  poles  along  the  roadway  marked  the  appropriate 
course.  Participants  were  instructed  to  leave  the  red  poles  on  their  right  and  to  complete  the 
exercise  without  collisions.  The  “slalom”  mission  required  operators  to  fly  a  slalom  course 
marked  out  by  six  red  and  green  poles.  Participants  were  instructed  to  leave  the  red  poles  on  their 
right  and  the  green  poles  on  their  left.  After  weaving  around  the  poles,  participants  had  to  turn 
1 80  degrees  and  fly  the  same  path  back  to  the  launch  point.  For  both  courses,  the  ideal  path  was 
displayed  on  the  situation  awareness  map,  and  participants  were  able  to  observe  at  all  times 
where  the  MAV  was  relative  to  the  path  drawn  on  the  map.  If  the  MAV  collided  with  an  obstacle 
(e.g.,  tree  or  building),  the  experimenter  stopped  the  trial  and  had  the  participant  start  the 
exercise  again  from  the  beginning.  Collisions  with  obstacles  halted  movement  in  the  direction 
the  MAV  was  traveling  at  the  time  of  the  collision  and  were  visibly  obvious.  Collisions  were 
also  recorded  automatically  in  log  files. 

Each  participant  was  allowed  up  to  five  attempts  to  successfully  complete  each  of  the 
seven  practice  exercises.  Participants  who  could  not  finish  an  exercise  within  five  attempts  were 
dismissed  from  the  experiment. 

After  finishing  the  practice  exercises,  participants  began  a  series  of  six  missions,  during 
which  the  main  performance  data  were  collected.  Participants  were  told  to  complete  each 
mission  as  quickly  as  possible  but  to  avoid  collisions.  There  were  no  performance  criteria  given 
to  participants,  no  waypoints  were  visible  on  the  situation  awareness  map,  and  missions 
continued  to  completion  whether  there  were  collisions  or  not.  In  order  to  continue  after  a 
collision,  participants  could  back  away  from  the  obstacle  and  then  resume  movement  around  it. 
Table  3  provides  a  summary  of  these  missions.  The  NASA  TLX  workload  questionnaire  was 
completed  by  participants  at  the  end  of  each  mission. 


Table  3 

Outline  of  missions.  Number  of  Collisions  Occurring  and  Self  rated  Workload  were  Recorded^ 


Mission 

Number 

Name  and  other 
Measures 

Description 

Mean 

Workload 

i 

Racetrack 

Time 

Maneuver  an  oval  route  with  the  aid  of  red  poles  placed 
to  guide  the  path;  land  on  a  rooftop. 

34.4 

2 

Church 

Time, 

Targets  located 

Free  flight  to  a  church  (identified  by  a  steeple);  inspect 
all  sides  and  determine  number  of  friendly  and  enemy 
inside. 

42.2 

3 

Slalom 

Time 

Maneuver  a  slalom  route  with  the  aid  of  red  and  green 
poles  (leave  red  on  right  and  green  on  left);  return  to 
launch  point. 

38.7 

4 

Reconnaissance 

Number  of  targets 
photographed  and 
recalled 

Free  flight  to  detect  and  photograph  targets  (Soldiers 
and  vehicles).  Pilot  recalled  to  launch  point  after  7 
minutes. 

49.8 

5 

Racetrack 

Time 

Same  as  Mission  1 

31.4 

6 

Slalom 

Time 

Same  as  Mission  3 

33.1 

11 


During  Mission  1,  participants  flew  the  racetrack  course  that  had  been  part  of  the  practice 
exercises;  however,  this  time,  the  ideal  path  to  follow  was  not  displayed  on  the  situation 
awareness  map. 

The  next  mission  was  the  first  of  two  tactical  missions  and  involved  flying  to  a 
designated  building  (a  church)  and  identifying  the  occupants,  as  seen  through  the  windows. 
Participants  were  briefed  on  their  mission  and  shown  pictures  of  friendly  and  enemy  Soldiers. 
They  were  instructed  to  observe  the  church  from  several  vantage  points  at  close  range  to 
determine  how  many  friendly  and  enemy  Soldiers  were  inside. 

During  Mission  3,  the  participants  flew  the  slalom  course  again;  however,  this  time,  the 
ideal  path  to  follow  was  not  displayed  on  the  situation  awareness  map. 

The  next  mission  was  a  reconnaissance  sortie.  Twelve  targets  (eight  vehicles  and  four 
dismounted  infantry)  were  imported  into  the  database.  Operators  received  a  mission  briefing 
illustrating  the  potential  targets.  They  were  instructed  to  fly  to  the  village  and  photograph  as 
many  of  the  targets  as  they  could  find  (there  were  no  distracters).  Each  target  was  to  be 
photographed  with  both  the  forward  and  downward  camera.  Participants  were  not  told  how  many 
targets  there  were  or  how  much  time  they  had  to  complete  the  mission.  After  seven  minutes  from 
launch,  operators  were  instructed  to  return  to  the  launch  point  immediately.  After  landing,  they 
were  given  a  debriefing  during  which  each  operator  was  asked  to  recall  the  type  and  location  of 
entities  observed  during  the  flight.  Participants  indicated  the  targets  they  could  recall  by  marking 
them  directly  on  a  scale  map  of  the  terrain.  This  post-test  was  later  graded  by  the  experimenter 
who  used  a  transparent  overlay  to  determine  correctly  marked  entity  locations.  Scores  were 
determined  as  follows:  correctly  identified  targets  located  within  .625”  of  the  actual  location  of 
the  entity  (inside  a  1 .25”  diameter  circle  traced  on  the  overlay)  received  2  points.  Entities  marked 
within  1”  of  the  actual  location  (within  a  2”  diameter  circle)  received  1  point.  Correctly  labeling 
an  entity’s  orientation  also  earned  a  point.  Orientation  for  vehicles  was  always  North. 

Participants  were  subsequently  instructed  to  repeat  the  racetrack  mission  and  then  the 
slalom  mission  (Missions  5  and  6).  They  then  completed  a  usability  survey  which  contained  3 1 
items  formatted  using  a  10-point  scale  with  1  indicating  the  most  favorable  response,  and  10  the 
least  favorable  response  (see  Appendix  F). 

Measures  and  Analysis 

Multiple  measures  of  performance  were  collected  for  each  mission,  with  some  variation 
depending  on  mission.  These  included  time-to-complete,  number  of  collisions,  number  of  targets 
observed,  and/or  photographed  (church  and  reconnaissance  missions),  and  self-rated  workload. 
Before  performing  any  formal  analysis,  data  were  screened  for  outliers  and  missing  cells. 

Outliers  were  treated  uniformly.  After  verifying  the  data  were  correctly  entered,  each  outlier 
score  was  transformed  to  one  measurement  unit  outside  of  the  most  extreme  score  that  fell  within 
+/-  2  SD  of  the  mean  (Tabachnick  &  Fidell,  1996).  There  were  two  instances  of  missing  data: 
one  male  and  one  female  participant  incorrectly  completed  a  single  spatial  test.  These  data  were 
replaced  by  using  the  mean  test  score  for  their  gender. 
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The  correlations  among  the  standardized  scores  on  the  spatial  tests  are  given  in  Table  4.  It 
can  be  seen  in  the  table  that  there  were  moderate  significant  correlations  among  the  different 
tests.  Examination  of  first-order  partial  correlations  indicated  that  none  of  the  test  scores  acted 
as  antecedents  or  interveners  in  the  relation  among  any  of  the  other  two.  This  suggests  that  the 
tests  measured  related  but  distinct  abilities.  To  control  for  individual  differences  when  assessing 
the  effects  of  the  independent  variables,  the  standardized  spatial  test  scores  were  summed  to 
create  a  single  covariate  for  analysis  of  covariance  (ANACOVA).  The  predictive  relation 
between  individual  spatial  test  scores  and  specific  measures  of  performance  was  further 
examined  if  the  composite  spatial  score  covaried  with  the  dependent  measure. 

Table  4 


Correlations  among  the  Spatial  AbilityTests 


Cube 

Comparison 

Hidden 

Figures 

Hidden  Patterns 

Spatial 

Orientation 

Cube 

Comparison 

.39  * 

.37  * 

.27  * 

Hidden 

Figures 

.42  * 

.15 

Hidden  Patterns 

.27  * 

Spatial 

Orientation 

*  p  <  .05 


Responses  to  the  three  questions  on  the  demographic  survey  relating  to  video-game 
experience  (VGE)  were  highly  correlated  and  combined  into  a  single  composite  measure.  This 
measure  was  correlated  significantly  with  gender  (Spearman  r  =  .75),  with  higher  video-game 
experience  associated  with  males.  The  composite  spatial  scores  were  also  correlated  with  gender 
(Spearman  r  =  .24),  with  higher  scores  more  likely  to  be  from  males.  Spatial  and  VGE  scores 
failed  to  be  significantly  correlated  (Spearman  r  =  .15).  The  eight  conditions  formed  by  the  2  x  2 
x  2  design  were  checked  for  equivalence  on  VGE  and  spatial  ability.  No  differences  across 
conditions  were  detected. 

The  effects  of  the  independent  variables  on  the  performance  measures  were  assessed 
using  ANACOVA,  with  the  spatial  ability  and  VGE  scores  as  covariates.  Exceptions  were  made 
for  number  of  collisions  and  number  of  attempts  (practice  exercises).  These  count  measures  were 
highly  positively  skewed  (i.e.,  mostly  0’s  for  collisions,  and  mostly  1  ’s  for  number  of  attempts). 
Consequently,  nonparametric  analysis  was  conducted  for  these  two  measures. 

Effects  reported  as  significant  reached  the  conventional  level  of/?  <  .05.  If  effects  are  not 
mentioned  (e.g.,  higher  order  interactions),  then  p  >  .05.  Effect  sizes  (</’)  for  significant  pair¬ 
wise  comparisons  are  reported.  This  measure  of  effect  size  represents  the  difference  between 
two  means  as  a  proportion  of  the  pooled  standard  deviation. 
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Results 


Practice  Exercise  Attempts 

The  practice  exercises  were  intended  to  train  operators  to  associate  input  controls  with 
their  corresponding  functions  and  to  navigate  designated  paths  through  the  virtual  environment. 
Table  5  shows  the  distribution  of  the  number  of  attempts  required  for  successful  completion  of 
each  exercise.  Both  spatial  ability  and  VGE  were  negatively  correlated  with  number  of  attempts 
required.  Higher  spatial  ability  and  greater  VGE  were  associated  with  fewer  required  attempts. 
For  the  first  five  practice  exercises,  in  which  participants  completed  a  series  of  directed 
maneuvers,  the  correlation  between  spatial  ability  and  number  of  attempts  was  significant 
(Spearman  r  =  -.30),  whereas  the  correlation  between  VGE  and  number  of  attempts  failed  to 
reach  significance  (Spearman  r  =  -.14).  For  the  racetrack  and  slalom  courses,  in  which 
participants  piloted  designated  paths,  both  measures  correlated  significantly  with  number  of 
attempts.  For  spatial  ability  Spearman  r  =  -.  27  and  for  VGE  Spearman  r  =  -.39.  The  maximum 
number  of  attempts  required  to  complete  all  7  of  the  practice  exercises  by  any  single  individual 
was  1 7.  No  significant  effects  of  the  independent  variables  on  number  of  attempts  were 
detected. 


Table  5 


Number  of  Participants  Requiring  1  to  5  Attempts  to  Complete  each  Practice  Exercise.  For 
Exercises  1,  2,  and  3Pparticipants  Read  a  List  of  Maneuvers.  For  Exercises  4 A  and  4B 
participants  Responded  to  a  List  of  Maneuvers  Said  Out  Loud  by  the  Experimenter.  The  Final 


Attempts: 

Practice  Exercise 

Criterion  Time 
(Minutes :  Seconds) 

1 

2 

3 

4 

5 

1  (read) 

0:30 

72 

0 

0 

0 

0 

2  (read) 

0:40 

72 

0 

0 

0 

0 

3  (read) 

1:35 

41 

24 

6 

0 

1 

4A  (oral) 

1:05 

50 

18 

3 

1 

0 

4B  (oral) 

1:25 

49 

17 

5 

0 

1 

Racetrack 

3:50 

47 

13 

7 

3 

2 

Slalom 

5:00 

39 

13 

12 

4 

4 

Racetrack  Practice  Criterion  Time 

Participants  were  required  to  navigate  the  racetrack  course  in  230s  or  less,  without  any 
collisions.  They  had  the  aid  of  an  ideal  path  (waypoints)  drawn  on  the  situation  awareness  map. 
There  failed  to  be  any  significant  relation  between  the  number  of  attempts  required  and  the 
actual  criterion  times  (Spearman  r  =  .1 1).  Neither  were  spatial  ability  nor  VGE  significant 
covariates  of  the  criterion  times.  Criterion  times,  which  are  listed  in  Table  6,  were  significantly 
affected  by  interactions  of  input  device  and  ICD,  F(l,  62)  =  5.81  and  input  device  and  number  of 
camera  views,  F(l,  62)  =  8.69;  the  three-way  interaction  failed  to  be  significant  (F  <  1).  As  can 
be  seen  in  the  table,  completion  tended  to  be  slower  in  the  mouse  conditions  (mean  207.1  s)  than 
the  game  controller  conditions  (mean  1 96.9  s);  however,  this  was  particularly  the  case  when  the 
mouse  was  combined  with  two  simultaneous  camera  views  (mean  21 1 .2  s)  or  with  the  discrete 
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ICD  (mean  212.6  s).  Analysis  of  the  mouse  conditions  indicated  main  effects  of  both  ICD, 

F(  1,  30)  =  7.74,  and  number  of  camera  views,  F(l,  30)  =  4.28.  In  contrast,  analysis  of  the  game 
controller  conditions  failed  to  suggest  any  significant  effects  of  these  variables. 


Table  6 

Mean  Time  in  Seconds  to  Complete  the  Racetrack  and  Slalom  Courses  on  the  Attempt  for  which 
the  Temporal  Criterion  (230  Secondsjor  Racetrack  and  300  Secondsjor  Slalom)  was  Mel 


Input  Device 

Camera  Views 

ICD 

Racetrack 

Criterion  Time 
Mean  (s.e.) 

Slalom  Criterion 
Time 

Mean  (s.e.) 

Mouse 

1 

Discrete 

209.4  (3.0) 

258.9(21.8) 

Mouse 

1 

Continuous 

196.7(4.2) 

230.7(35.1) 

Mouse 

2 

Discrete 

215.8(3.3) 

243.9  (23.3) 

Mouse 

2 

Continuous 

206.5  (4.6) 

234.7  (26.3) 

Game-Controller 

1 

Discrete 

196.8  (4.4) 

225.6  (29.8) 

Game-Controller 

1 

Continuous 

201.8  (4.5) 

226.1  (16.6) 

Game-Controller 

2 

Discrete 

191.2  (3.7) 

215.8(21.9) 

Game-Controller 

2 

Continuous 

197.6  (4.9) 

227.3  (25.8) 

Racetrack  Missions 

Figure  5  illustrates  the  mean  time  to  complete  the  racetrack  missions  for  the  two  times  it 
was  completed  during  the  mission  phase  of  the  experiment  (Missions  1  and  5).  During  the 
mission  phase,  there  was  no  time  criterion  and  the  run  went  to  completion  whether  there  was  a 
collision  or  not.  No  path  was  displayed  on  the  situation  awareness  map.  VGE  and  spatial  ability 
failed  to  be  significant  covariates  of  mission  time.  It  can  be  seen  in  the  figure  that  mission  time 
decreased  from  the  first  to  the  second  repetition,  F(l,  62)  =  15.90,  d'  =  .43.  There  was  also  an 
input  device  by  ICD  interaction,  F(l,  62)  =  15.9.  In  the  discrete  ICD  conditions,  participants 
using  the  mouse  were  significantly  slower  to  complete  the  course  than  participants  using  the 
game  controller.  In  the  continuous  ICD  conditions,  there  failed  to  be  any  difference  due  to  input 
device.  According  to  post  hoc  tests  (Tukey  HSD),  participants  in  the  mouse-plus-discrete  ICD 
were  significantly  slower  than  participants  in  any  of  the  other  conditions. 
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Figure  5.  Mean  seconds  to  complete  the  racetrack  missions  for  its  two  repetitions  (first  and 
second).  The  left  panel  shows  the  results  for  the  discrete  ICD  condition,  and  the  right  panel  for 
the  continuous  ICD  condition.  Vertical  bars  show  95%  confidence  intervals. 

Seven  participants  had  at  least  one  collision  during  the  first  repetition  of  the  racetrack 
mission,  and  1 1  had  at  least  one  collision  during  the  second  repetition.  None  of  these  were  the 
same  participants.  Some  participants  expressed  determination  to  better  their  completion  time  on 
the  second  repetition  (even  though  they  were  not  instructed  to  do  so).  Consequently,  they  may 
have  taken  more  risk  in  flight  than  was  prudent.  There  failed  to  be  a  systematic  pattern  between 
condition  and  collisions.  Neither  was  there  any  significant  relationship  detected  between 
collisions  and  spatial  ability  or  VGE. 

Mean  subjective  workload  was  32.9  for  the  racetrack  mission,  averaged  across  conditions 
and  repetition.  There  failed  to  be  any  detectable  effect  of  condition  or  repetition;  however,  VGE 
was  a  significant  covariate  of  workload  scores,  F(l,  62)  =  1 1.04.  Higher  VGE  was  associated 
with  lower  workload  scores. 

Slalom  Practice  Criterion  Time 

Participants  were  required  to  navigate  the  slalom  course  in  300  seconds  or  less,  without 
any  collisions.  They  had  the  aid  of  an  ideal  path  drawn  on  the  situation  awareness  map.  There 
failed  to  be  any  significant  relation  between  the  number  of  attempts  required  and  the  actual 
criterion  times  (Spearman  r  =  .  1 6).  VGE  was  a  significant  covariate  of  completion  time,  F(  1 ,  62) 
=  5.02,  with  higher  VGE  associated  with  lower  times.  As  can  be  seen  in  Table  6,  mean  criterion 
times  were  higher  for  participants  using  the  mouse  (mean  241 .7  s)  than  the  game  controller 
(mean  223.0  s);  however  there  was  an  interaction  of  input  device  and  ICD,  which  modulated  this 


16 


difference,  F{\,62)  =  4.8.  Analysis  of  the  mouse  conditions  indicated  a  significant  effect  of  ICD, 
F(l,  30)  =  4.309,  with  those  in  the  discrete  condition  performing  more  slowly.  For  the  game- 
controller  conditions,  there  failed  to  be  an  effect  of  ICD. 

Slalom  Missions 

Figure  6  illustrates  the  mean  time  to  complete  the  slalom  missions  for  the  two  times  it 
was  completed  during  the  mission  phase  of  the  experiment.  During  the  mission  phase,  there  was 
no  time  criterion,  and  the  run  went  to  completion  whether  there  was  a  collision  or  not.  No  path 
was  displayed  on  the  situation  awareness  map.  VGE  and  spatial  ability  failed  to  be  significant 
covariates  of  mission  time.  It  can  be  seen  in  the  figure  that  mission  time  decreased  from  the  first 
to  the  second  repetition,  F(l,  62)  =  44.31.  It  can  also  be  seen  that  participants  using  the  mouse 
were  slower  to  complete  missions  than  participants  using  the  game  controller,  F(l,  62)  =  12.55. 
There  was  also  an  interaction  of  input  device  and  repetition,  F(l,  62)  =  4.36.  The  differences  due 
to  input  device  were  greater  on  the  first  than  the  second  repetition.  A  post  hoc  test  (Tukey  HSD) 
indicated  that  within  each  input  condition  there  were  significant  time  decreases  from  repetition 
one  to  repetition  two  ( d '  =  1.30  and  .45  for  mouse  and  game  controller  conditions,  respectively). 
Across  conditions,  input  device  had  a  significant  effect  during  repetition  one  ( d '  =  .84),  but  failed 
to  do  so  during  repetition  two.  The  interaction  of  input  device  and  ICD,  which  was  found  for  the 
slalom  practice,  failed  to  be  significant  for  the  slalom  missions. 


Figure  6.  Mean  seconds  to  complete  the  slalom  missions  for  its  two  repetitions  (first  and  second). 
The  left  panel  shows  the  results  for  the  discrete  ICD  condition,  and  the  right  panel  for  the 
continuous  ICD  condition.  Vertical  bars  show  95%  confidence  intervals. 
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Sixteen  participants  had  at  least  one  collision  during  the  first  repetition  of  the  slalom 
mission,  and  1 6  had  at  least  one  during  the  second  repetition.  Of  these,  six  had  collisions  during 
both.  There  failed  to  be  a  systematic  pattern  between  condition  and  collisions.  Neither  was  there 
any  significant  relationship  detected  between  collisions  and  spatial  ability  or  VGE. 

VGE  was  a  significant  covariate  of  workload  ratings  for  the  slalom  mission,  F(l,  62)  = 
5.40,  with  higher  VGE  associated  with  lower  workload  ratings.  Mean  workload  for  the  first  and 
second  slalom  repetitions  was  38.7  and  33.1,  respectively.  This  decrease  was  significant, 

FO,  62)  =  25.21,  d'  =  .26;  but  the  effect  of  repetition  interacted  with  both  input  device, 

F(l,  62)  =  4.70,  and  ICD,  F(l,  62)  =  9.90.  With  respect  to  input  device,  a  post  hoc  analysis 
indicated  that  the  decrease  was  significant  only  in  the  mouse  conditions  (40.5  to  32.5,  d'  =  .38). 
With  respect  to  the  ICD,  a  post  hoc  analysis  indicated  that  the  decrease  was  significant  only  in 
the  continuous  conditions  (38.29  to  29.35,  d'  =  .45). 

Church  Mission 

This  mission  required  participants  to  locate  the  church  and  inspect  it  from  all  sides  to 
determine  who  was  inside  (two  friendly  and  two  enemy  Soldiers).  Only  48  of  the  72  participants 
accurately  reported  this.  Those  who  were  inaccurate  failed  to  inspect  the  church  from  all  sides, 
and  consequently  had  much  shorter  times  to  complete  the  mission.  Because  of  this,  only  data 
from  the  48  accurate  participants  were  analyzed.  The  number  of  accurate  participants  in  each  of 
the  eight  conditions  ranged  between  three  and  nine,  and  was  not  systematically  related  to 
condition. 

VGE  was  a  significant  covariate  of  time  to  complete  the  church  mission,  F(l,  38)  =  8.30, 
with  greater  VGE  associated  with  shorter  times.  Figure  7  illustrates  the  effect  of  input  device  and 
number  of  camera  views  on  time  to  complete  the  church  mission.  Participants  completed  the 
mission  faster  with  the  game  controller  (N  =  27)  than  the  mouse  (N  =  21 ),  F(l,  38)  =  12.90,  d ’  = 
.97.  They  also  completed  the  mission  faster  with  only  one,  as  compared  with  two  camera  views 
(N=  24  for  each  condition),  F(l,  38)  =  7.17,  d'  =  .64. 


18 


Number  of  Simultaneous  Camera  Views 


Figure  7.  Mean  seconds  to  complete  the  church  mission,  according  to  the  input  device  and  the 
number  of  simultaneous  camera  views.  Vertical  bars  show  95%  confidence  intervals. 

Only  eight  of  the  27  participants  using  the  game  controller  collided  with  an  obstacle 
during  the  church  mission,  whereas  13  of  the  21  participants  using  the  mouse  did  so.  This 
difference  was  significant  according  to  a  two-tailed  Fisher  exact  test.  The  other  independent 
variables  failed  to  affect  collisions.  VGE  correlated  significantly  with  number  of  collisions 
(Spearman  r  =  -.36). 

VGE  was  a  significant  covariate  of  workload  ratings,  F(l,  38)  =  6.67,  with  higher  VGE 
associated  with  lower  workload  scores.  Participants  rated  workload  higher  if  using  the  mouse 
(mean  44.6)  than  if  using  the  game  controller  (mean  32.5),  F(l,38)  =  5.44,  d'  =  .32. 

Reconnaissance  Mission 

For  number  of  targets  photographed,  a  full  point  was  awarded  for  each  target 
photographed  with  both  the  forward  and  downward  cameras,  and  Vi  point  for  each  target 
photographed  with  only  one  of  the  cameras  (no  extra  points  were  awarded  for  additional  photos 
of  a  target  once  one  forward  and  one  downward  picture  were  taken).  Therefore,  the  maximum 
possible  score  was  twelve.  Both  VGE  and  spatial  ability  were  significant  covariates  of  number  of 
targets  photographed.  For  VGE,  F(l,  62)  =10.10,  and  for  spatial  ability,  F(l,62)  =  16.26. 

Figure  8  shows  the  mean  number  of  targets  photographed,  according  to  input  device  and 
ICD.  Participants  using  the  game  controller  photographed  significantly  more  targets  than  those 
using  the  mouse,  F(l,  62)  =  28.44,  d'  =  .95.  There  was  also  an  interaction  of  input  device  and 
ICD,  F(l,  62)  =  5.23.  The  combination  of  the  mouse  and  the  continuous  ICD  yielded  the  lowest 
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mean  number  of  photographs,  and  according  to  a  Tukey  HSD  test,  this  condition  differed 
significantly  from  both  game  controller  conditions,  and  marginally  significantly  (adjusted  p  < 
.06)  from  the  mouse  and  discrete  ICD  condition.  For  this  latter  comparison,  d ’  =  .74. 
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Figure  8.  Mean  number  of  targets  photographed  during  the  reconnaissance  mission,  according  to 
input  device  and  ICD.  Vertical  bars  show  95%  confidence  intervals. 

Subsequent  to  landing,  each  participant  was  asked  to  recall  the  type  and  location  of 
entities  observed  during  the  flight  by  marking  target  locations  on  a  scale  map  of  the  terrain.  In 
order  to  receive  one  point,  the  target  type  had  to  be  correctly  indicated  and  fall  within  a  2” 
diameter  circle  around  the  actual  target  location.  Correct  identifications  within  a  1 .25”  diameter 
circle  received  two  points.  Correctly  labeling  an  entity’s  orientation  earned  an  additional  point 
(all  targets  were  oriented  north).  Thus,  the  maximum  possible  score  on  this  measure  was  36 
(three  points  per  target).  The  mean  target  recall  score  was  12.7  (standard  deviation  =  5.4).  None 
of  the  independent  variables  had  a  significant  effect  on  recall  scores;  however,  both  VGE  and 
spatial  ability  were  significant  covariates.  For  VGE,  F(l,  62)  =  9.56,  and  for  spatial  ability, 

F(l,  62)  =  4.59. 

Forty-six  of  the  72  participants  committed  at  least  one  collision  during  the  reconnaissance 
mission.  Median  number  of  collisions  was  1 .0.  Whether  a  participant  had  a  collision  or  not  was 
unrelated  to  condition  statistically;  however,  there  was  a  significant  relationship  between  spatial 
ability  and  number  of  collisions  (Spearman  r  =-25).  Higher  spatial  ability  tended  to  be  associated 
with  fewer  collisions. 
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VGE  was  a  significant  covariate  of  workload,  F(  1,  62)  =  7.0,  with  higher  VGE  associated 
with  lower  rated  workload.  The  independent  variables  failed  to  significantly  affect  workload 
during  this  mission. 

Relation  between  Individual  Spatial  Ability  Tests  and  the  Dependent  Variables 

According  to  the  analyses  above,  spatial  ability  correlated  with  number  of  attempts  to 
complete  the  practice  exercises,  number  of  targets  photographed  and  recalled,  and  number  of 
collisions  in  the  reconnaissance  mission.  Although  not  the  main  purpose  of  this  experiment,  we 
evaluated  whether  the  individual  spatial  tests  differed  in  their  ability  to  predict  variation  in  these 
dependent  variables.  For  number  of  attempts  to  complete  practice  exercises  and  collisions,  we 
examined  Spearman  correlations  because  the  data  did  not  conform  to  the  assumptions  required 
for  parametric  analysis.  Table  7  shows  the  Spearman  correlations  between  the  individual  spatial 
ability  tests  and  these  dependent  variables.  For  the  directed  maneuver  exercises  (practice 
exercises  1-5),  there  were  significant  correlations  between  the  Cube  Comparison  test  and  the 
Hidden  Patterns  test  with  number  of  attempts  required.  For  the  racetrack  course,  there  were 
significant  correlations  between  the  Hidden  Patterns  test  and  the  Hidden  Figures  test  with 
number  of  attempts  required.  For  the  slalom  course,  there  failed  to  be  any  significant 
correlations.  For  the  reconnaissance  mission,  there  was  a  significant  correlation  between 
collisions  and  the  Hidden  Patterns  test. 


Table  7 

Spearman  Correlations  between  Dependent  Variables  and  the  Individual  Spatial  Ability  Tests 


Directed 

Maneuver 

Attempts 

Racetrack 

Attempts 

Slalom 

Attempts 

Recon  Mission 
Collisions 

Cube  Comparison 

-.27* 

-.14 

.00 

-.19 

Hidden  Figures 

-.10 

-.34* 

-.07 

-.14 

Hidden  Patterns 

-.35* 

-.27* 

-.01 

-.32* 

Spatial  Orientation 

-.10 

-.12 

-.14 

-.13 

*  p  <  .05 


For  the  number  of  targets  photographed  during  the  reconnaissance  mission,  we  conducted 
an  ANACOVA  with  VGE  and  all  four  spatial  tests  as  covariates,  and  the  three  manipulated 
factors  (input  device,  ICD,  and  number  of  camera  views).  According  to  this  analysis  of  the  four 
spatial  tests,  only  the  Hidden  Patterns  test  was  a  significant  covariate,  F(l,  59)  =  4.93.  An 
analogous  ANACOVA  for  number  of  targets  recalled  (drawn  on  the  map  post-mission)  indicated 
that  none  of  the  individual  spatial  tests  were  significant  covariates. 

In  summary,  the  Hidden  Patterns  test  had  the  most  consistent  ability  to  account  for 
variation  in  these  dependent  variables.  For  two  of  the  analyses,  no  individual  tests  correlated 
significantly  with  the  dependent  variable  examined  despite  significant  correlations  between  the 
composite  spatial  measure  and  the  dependent  variables. 
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Usability 


Overall,  usability  was  rated  more  favorably  by  participants  who  used  the  game  controller 
than  by  participants  who  used  the  mouse,  F(1 ,  62)  =  6. 16,  d.'  =  .56.  Averaged  over  all  the  3 1 
questions,  the  mean  usability  score  was  2.3  for  the  game  controller  condition  and  2.8  for  the 
mouse  condition  (with  lower  scores  indicating  more  favorable  ratings).  The  extent  of  the 
difference  caused  by  input  device  depended  on  the  individual  questions.  There  was  a  question  by 
input  device  interaction,  F(l,  1860)  =  3.16.  The  ratings  from  the  game  controller  condition  were 
significantly  more  favorable  than  from  the  mouse  condition  for  nine  questions.  These  included 
ratings  of  the  system  overall,  system  feedback,  ease  of  controlling  the  MAV,  and  the  ability  to 
maintain  awareness  of  mission  objectives.  The  specific  data  associated  with  these  questions  are 
listed  in  Table  8. 

Table  8 


Mean  Usability  Questions  for  which  Participants  Using  the  Game  Controller  (GC)  Gave 
Significantly  More  Favorable  Ratings  (Lower  Scores)  than  Participants  Using  the  Mouse,  along 
with  Corresponding  Statistics 


Usability  Question 

Mean 

Mouse 

Mean 

GC 

F(  1,62) 

d' 

The  system  I  worked  with  was 

1  Easy - Difficult  10 

3.3 

2.6 

8.97 

.69 

When  controlling  the  MAV  in  flight  using  the  input  device 

was 

1  Easy - Difficult  10 

4.0 

2.3 

46.72 

.78 

When  using  the  input  device  to  enter  flight  commands  to 

the  MAV,  maintaining  awareness  of  individual  mission 
objectives  was 

1  Easy - Difficult  10 

3.3 

2.2 

9.01 

.69 

When  giving  commands  to  move  forward,  back,  left  & 

right,  the  air  vehicle  reacted  as  I  expected. 

1  Always - Never  10 

3.6 

2.4 

12.01 

.76 

When  giving  commands  to  move  up,  down  &  rotate,  the 

air  vehicle  reacted  as  I  expected. 

1  Always - Never  10 

3.6 

2.5 

6.74 

.60 

The  system  provided  adequate  feedback  when  I  issued  a 

command  to  the  MAV. 

1  Always - Never  10 

2.7 

2.0 

5.76 

.54 

The  OCU  interface  keeps  you  informed  about  what  it  is 
happening. 

1  Always - Never  10 

2.7 

2.0 

6.41 

.57 

Correcting  your  mistakes  was 

1  Easy - Difficult  1 0 

3.4 

2.6 

4.48 

.49 

Both  experienced  and  inexperienced  users’  needs  were 

taken  into  consideration. 

1  Always - Never  10 

2.5 

1.8 

5.07 

.53 
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There  were  only  three  questions  (out  of  31)  for  which  the  scores  were  reversed,  and  only  for  one 
of  these  was  the  difference  significant.  This  question  was  “System  speed  was,”  with  options  1  to 
10  where  1  was  “fast  enough”  and  10  was  “too  slow.”  Mean  ratings  for  this  question  were  3.2 
and  4.7  for  the  mouse  and  game  controller  conditions,  respectively,  F(  1,  62)  =  6.04,  d'  =  .56.  We 
had  intended  this  question  to  be  about  the  responsiveness  of  the  OCU,  but  many  participants 
asked  if  this  was  about  the  speed  of  the  MAV.  They  were  instructed  to  make  their  own 
interpretation.  The  results  on  this  question,  therefore,  indicate  that  game  controller  users  were 
more  likely  to  think  they  could  handle  a  faster  vehicle  or  a  more  responsive  OCU  than  the  mouse 
users  did. 

With  respect  to  VGE  and  spatial  ability,  there  was  a  lack  of  parallelism  across  usability 
questions.  In  other  words,  these  measures  correlated  with  usability  responses  for  some 
questions,  but  not  others.  The  interactions  between  question  and  VGE,  and  question  and  spatial 
ability  were  significant,  F’s  (1,  1860)  =  1.72  and  1.97,  respectively.  Table  9  lists  the  five 
questions  for  which  one  or  both  of  the  measures  correlated  significantly,  along  with  the 
Spearman  correlations.  Higher  spatial  ability  was  associated  with  less  frustration  and  confusion, 
and  greater  confidence  about  flying  a  real  vehicle.  Higher  VGE  was  associated  with  higher 
perceived  ease  of  use  and  greater  confidence  about  flying  a  real  vehicle;  but  also  with  less 
satisfaction  with  status  messages. 

There  was  a  significant  positive  association  between  average  usability  ratings  and 
average  workload  ratings.  Lower  workload  ratings  were  associated  with  more  favorable 
usability  ratings  (Spearman  r  =  .50). 


Table  9 

Usability  Questions  which  Produced  Ratings  Significantly  Correlated  with  VGE  or  Spatial 
Ability  Along.  the  Spearman  Correlations _ 


Usability  Question 

VGE 

Spatial 

Ability 

The  system  I  worked  with  was 

1  Easy - Difficult  1 0 

-.24* 

-.22 

I  found  this  experience 

1  Satisfying - Frustrating  10 

-.17 

-.31* 

Current  status  (such  as  Taking  off,  Landing,  Grounded) 

messages  were 

1  Adequate - Inadequate  10 

.30* 

-.01 

When  piloting  the  MAV  in  manual  control,  determining  the 

current  heading  of  the  MAV  from  the  360-degree 
directional  heading  tape  was 

1  Clear - Confusing  1 0 

-.19 

-.29* 

After  this  training,  1  am  ready  use  this  system  to  fly  a  real 

air  vehicle 

1  Very  confident - Not  at  all  confident  10 

-.39* 

-.27* 

*  indicates  significance  at  the  .05  level.  A  negative  correlation  indicates  that  higher  VGE  or 
spatial  ability  corresponds  to  more  favorable  usability  ratings. 


23 


Discussion  &  Conclusions 


MAVs  and  NAVs  have  limited  battery  and/or  fuel  capacity.  Consequently,  the  ability  of 
operators  to  complete  missions  as  quickly  as  possible  is  an  important  skill.  According  to  the 
present  data,  time  to  complete  missions  was  affected  by  input  device  or  an  interaction  of  input 
device  and  ICD.  In  general,  missions  took  longer  to  complete  with  the  mouse,  especially  when 
the  mouse  was  combined  with  the  discrete  ICD.  Times  for  the  mouse-plus-continuous  ICD 
condition  produced  completion  times  similar  to  the  game  controller  conditions  in  some  cases 
(e.g.,  racetrack  course),  but  similar  to  the  mouse-plus-discrete  ICD  condition  in  other  cases  (e.g., 
church  mission).  The  specific  pattern  may  have  depended  on  mission  difficulty.  When  workload 
was  rated  relatively  low  (e.g.,  racetrack),  presumably  the  mission  was  relatively  easy; 
participants  in  the  mouse-plus-continuous  ICD  condition  were  able  to  complete  the  mission  as 
quickly  as  participants  with  the  game  controller.  When  workload  was  rated  relatively  high  (e.g., 
church  mission),  presumably  the  mission  was  relatively  challenging.  Participants  in  the  mouse- 
plus-continuous  ICD  were  unable  to  complete  the  mission  as  quickly  as  participants  with  the 
game  controller.  Instead,  their  performance  resembled  that  of  the  mouse-plus-discrete  ICD 
group.  Notably  for  the  church  mission,  workload  was  rated  significantly  lower  by  participants 
using  the  game  controller  than  by  participants  using  the  mouse.  Also  for  the  church  mission,  the 
number  of  collisions  was  significantly  lower  by  participants  using  the  game  controller  than  by 
participants  using  the  mouse. 

The  pattern  suggests  that  missions  were  most  efficiently  conducted  with  the  game 
controller,  least  efficiently  conducted  with  the  mouse-plus-discrete  ICD,  and  that  the  mouse- 
plus-continuous  ICD  was  intermediate  to  these.  However,  this  generalization  does  not  fit  with 
the  reconnaissance  mission  data.  All  participants  were  given  the  same  amount  of  time  to 
complete  the  reconnaissance  mission  and  the  key  data  were  number  of  targets  photographed. 
While  more  targets  were  photographed  by  game  controller  users  than  by  mouse  users,  there  was 
an  interaction  of  input  device  and  ICD  such  that  mouse  users  with  the  discrete  ICD  managed  to 
photograph  more  targets  than  mouse  users  with  the  continuous  ICD.  This  poor  performance  with 
the  mouse-plus-continuous  ICD  (relative  to  the  mouse  and  discrete  ICD)  does  not  fit  with  the 
simple  characterization  given  above,  but  might  be  understood  with  some  reflection  on  the 
mission  requirements.  In  order  to  photograph  targets,  participants  had  to  line  up  the  target 
indicator  on  the  target  by  maneuvering  the  MAV  (because  the  cameras  were  fixed).  A  common 
strategy  was  to  bring  the  MAV  to  a  hover  with  the  target  indicator  near  the  target,  and  then  to 
make  fine-grained  movements  to  get  the  indicator  lined  up  on  the  target  before  taking  the  picture. 
With  the  game  controller  the  MAV  could  be  put  into  a  hover  simply  by  releasing  the  thumb 
switches.  With  the  discrete  ICD,  the  MAV  could  be  put  into  a  hover  simply  by  clicking  on  the  X. 
In  contrast,  with  the  continuous  ICD,  the  command  to  hover  was  more  demanding.  All  the  dots 
had  to  be  returned  to  their  home  base,  so  multiple  actions  may  have  been  required  to  put  the 
MAV  into  a  hover.  It  is  possible  that  extra  actions  required  in  the  mouse-plus-continuous  ICD 
condition  account  for  the  fewer  photographs  taken  in  this  condition. 

Consideration  of  the  time  data  and  the  reconnaissance  mission  data  together  suggests  a 
rather  common  sense  rule  of  thumb  for  designing  discrete  vs.  continuous  interface  control 
schemes.  The  control  method  should  match  the  function  of  the  command.  A  discrete  command 
(like  hover)  is  likely  best  implemented  as  a  single  discrete  command.  Inputs  that  control 
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continuous  movements  through  space,  on  the  other  hand,  are  likely  best  implemented  as 
continuous  commands.  A  good  method  for  making  the  MAV  stop  is  not  simply  “undoing”  what 
makes  the  MAV  go,  as  illustrated  in  the  mouse-plus-continuous  ICD  condition. 

Besides  the  continuous  control  of  maneuver  that  the  game  controller  afforded,  it  may 
have  provided  operators  with  another  advantage.  Once  the  functions  of  the  game  controller 
buttons  are  learned,  the  user  has  little  need  to  remove  attention  from  the  sensor  imagery.  In 
contrast,  for  a  point  and  click  method,  such  as  a  mouse  or  a  touch-screen,  the  user  has  to  divert 
attention  from  the  sensor  imagery  to  the  ICD  or  other  iconology  every  time  a  new  instruction  is 
issued.  This  is  because  users  have  to  ensure  they  point  to  the  right  location  for  the  intended 
command.  Any  input  method  which  allows  for  focused  visual  attention  on  the  sensor  imagery, 
instead  of  one  which  requires  frequent  shifts  of  visual  attention  away  from  it,  will  likely  produce 
superior  performance.  More  focused  attention  on  the  sensor  imagery  may  improve  performance 
for  several  reasons.  First,  it  may  simply  be  more  efficient.  Less  time  is  required  because  imagery 
interpretation  and  command  execution  can  be  performed  in  parallel.  Second,  it  may  be  less 
cognitively  taxing  than  having  to  repeatedly  re-orient  visual  attention  from  one  part  of  the 
display  to  another  (Rubinstein,  Meyer,  &  Evans,  2001;  Wickens  &  Liu,  1988).  Third,  it  may 
lessen  opportunities  for  change  blindness,  i.e.,  the  tendency  to  miss  imagery  changes  occurring 
while  attention  is  diverted  to  other  aspects  of  the  OCU  (Durlach,  2004;  Durlach  and  Meliza, 
2004).  Fourth,  it  may  facilitate  the  operator’s  processing  of  and  learning  about  features  in  the 
environment  in  support  of  navigation  and  collision  avoidance,  such  as  the  spatial  relationships 
among  landmarks. 

The  manipulation  of  one  vs.  two  camera  views  did  not  have  the  expected  impact  on 
performance.  Our  original  hypothesis  was  that  two  views  might  improve  performance  by 
allowing  better  spatial  awareness  and  eliminating  the  task  of  view  switching.  Contrary  to  these 
expectations,  when  there  was  a  significant  effect  of  camera  views,  performance  was  impaired  in 
the  two-view  condition  (e.g.,  church  mission  time).  This  might  have  occurred  because  the 
simultaneous  views  took  up  the  same  screen  area  as  the  single  view.  Consequently,  each  image 
was  smaller,  and  this  might  have  had  negative  effects  on  piloting  and/or  target  detection  (e.g., 
Stelzer  &  Wickens,  2005;  Wickens,  2005).  The  church  mission  may  have  been  particularly 
sensitive  to  these  effects  because  it  required  careful  maneuvering  around  the  church  as  well  as 
target  identification  (friend  or  foe). 

VGE  and  spatial  ability  were  measured  in  order  to  control  for  participant  differences  in 
these  variables  and  therefore  more  clearly  reveal  the  effects  of  the  independent  variables. 
Nevertheless,  the  relation  among  VGE,  spatial  ability,  and  the  performance  measures  are  of  some 
interest  in  their  own  right.  Higher  VGE  was  associated  with  faster  mission  completion,  fewer 
collisions,  more  photographs  taken,  more  targets  recalled  and  lower  workload  scores.  This  is 
hardly  surprising  because,  essentially,  the  MAV  simulation  is  a  video  game.  It  has  been 
suggested  that  VGE  may  result  in  changes  in  visual  attention  (Green  &  Bavelier,  2003).  Such 
changes  might  account  for  the  association  between  VGE  and  better  performance.  On  the  other 
hand,  such  effects  have  not  been  easily  replicated  (Castel,  Pratt,  &  Drummand,  2005;  Durlach, 
Kring,  &  Bowens,  in  press).  It  may  be  that  higher  VGE  is  simply  associated  with  greater 
acceptance  of  and  comfort  in  simulation  environments.  Higher  VGE  was  associated  with  better 
performance  regardless  of  input  device.  Thus,  it  was  not  merely  due  to  greater  familiarity  with 


25 


the  game  controller  device,  but  rather  to  some  general  effect  on  operating  in  a  simulation 
context.  It  is  unknown  whether  this  effect  will  generalize  to  operation  of  a  real  (as  opposed  to 
simulated)  MAV  system. 

Higher  spatial  ability  was  associated  with  the  need  for  fewer  attempts  to  complete  the 
practice  exercises.  Spatial  ability  also  accounted  for  some  of  the  variation  in  performance  on  the 
reconnaissance  mission.  For  the  reconnaissance  mission,  higher  spatial  ability  was  associated 
with  fewer  collisions,  more  targets  photographed,  and  more  targets  recalled.  The  reconnaissance 
mission  was  rated  the  most  demanding,  in  terms  of  workload;  so,  it  may  be  that  the  other 
missions  failed  to  reveal  an  impact  of  spatial  ability  because  they  were  not  challenging  enough. 
In  terms  of  usability  ratings,  higher  spatial  ability  was  associated  with  less  frustration  and 
confusion,  and  greater  confidence  about  flying  a  real  vehicle.  With  respect  to  the  four  spatial 
ability  tests  used,  the  hidden  patterns  test  appeared  to  be  most  predictive;  however,  there  were 
instances  where  the  composite  spatial  ability  score  significantly  correlated  with  performance,  but 
each  individual  test  failed  to  do  so. 

Besides  evaluating  the  impact  of  input  device,  ICD,  and  number  of  camera  views,  one  of 
the  purposes  of  this  research  was  to  design  simulated  missions  that  could  be  used  for  operator 
training  and  to  suggest  measures  which  could  be  used  to  track  the  course  of  operator  mastery. 
Our  intention  was  to  train  all  the  elements  of  control  that  an  operator  might  need  in  manual 
mode.  In  order  to  do  this,  we  devised  what  we  considered  to  be  increasingly  difficult  exercises 
and  missions.  We  included  repetitions  of  the  racetrack  and  slalom  courses  in  order  to  provide 
evidence  of  improvement.  From  the  present  data  it  is  not  really  possible  to  describe  the  source  of 
the  improvement  observed,  however.  It  may  have  been  due  to  mastery  of  flight  controls  and 
improved  understanding  of  the  simulated  MAV’s  flight  dynamics.  It  also  may  have  been  due  to 
greater  familiarity  with  the  simulated  environment.  Because  all  the  exercises  were  conducted  in 
the  same  environment,  participants  had  the  opportunity  to  learn  the  location  of  obstacles  (e.g., 
trees),  the  layout  of  the  town,  and  useful  landmarks  to  guide  their  flight.  It  is  not  clear  to  what 
extent  learning  about  the  spatial  layout  of  the  environment  accounted  for  improved  mission 
performance.  To  separate  the  effects  of  learning  actual  flight  control  from  learning  about 
environmental  cues,  performance  must  be  tested  in  a  novel  environment. 

The  participants  in  this  experiment  had,  on  average,  two  hours  of  training,  so  it  is 
possible  that  performance  could  have  improved  even  more.  The  fact  that  some  participants  had 
collisions  during  the  last  mission  attests  to  this.  It  is  possible  that  a  greater  amount  of  training 
could  alter  some  of  the  effects  observed  in  the  experiment.  For  example,  with  more  experience 
operators  might  actually  benefit  from  simultaneous  camera  views.  Multiple  views  might  also 
have  proved  advantageous  had  we  tested  in  a  novel  environment.  These  possibilities  can  only  be 
addressed  with  additional  research. 

While  two  hours  of  manual  mode  training  does  not  seem  like  a  lot,  it  may  not  be  very  far 
out  of  line  with  what  operators  will  actually  receive.  Manual  mode  training  is  just  one  of  the 
many  elements  of  MAV  employment  that  requires  training  (Durlach,  2007).  For  example, 
operators  must  be  trained  on  vehicle  assembly,  maintenance,  power,  communications,  terrain  and 
weather  effects,  waypoint  navigation,  mission  planning,  and  tactics.  With  limited  time  available 
for  training,  it  is  important  that  training  be  designed  to  be  as  efficient  and  effective  and  that  the 
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OCU  be  designed  to  provide  the  affordances  that  will  best  support  the  operator’s  tasks.  The 
present  results  suggest  a  fairly  common  sense  heuristic:  continuous  control  for  continuous 
functions  and  discrete  control  for  discrete  functions. 
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Appendix  A 
Acronyms 


ANACOVA 

Analysis  of  Covariance 

ARI 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

DARPA 

Defense  Advanced  Research  Projects  Agency 

ETS 

Educational  Testing  Service 

1CD 

Input  Control  Display 

MAV 

Micro- Aerial  Vehicle 

NAV 

Nano-Aerial  Vehicle 

ocu 

Operator  Control  Unit 

RSTA 

Reconnaissance,  Surveillance,  and  Target  Acquisition 

UAS 

Unmanned  Aviation  Systems 

VGE 

Video  Game  Experience 
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Appendix  B 

Illustrations  of  Operator  Control  Unit  Display  Configurations 


Figure  B-l .  OCU,  single  view  overlapping  cameras  with  discrete  input  control . B-l 

Figure  B-2.  OCU,  single  view  overlapping  cameras  with  continuous  input  control . B-2 

Figure  B-3.  OCU,  two  adjacent  camera  views  with  discrete  input  control . B-3 

Figure  B-4.  OCU,  two  adjacent  camera  views  with  continuous  input  control . B-4 
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Figure  B-l.  OCU,  single  view  camera  layout  with  discrete  input  control. 
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Figure  B-2.  OCU,  single  view  camera  layout  with  continuous  input  control. 
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Figure  B-3.  OCU,  two  simultaneous  camera  views  with  discrete  input  control. 
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Figure  B-4.  OCU,  two  simultaneous  camera  views  with  continuous  input  control. 
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Appendix  C 

Participant  Training  Guide 

A  different  training  manual  was  produced  for  each  of  the  eight  conditions.  To  create  these  an 
original  “baseline”  guide  was  edited  to  take  into  account  the  required  variations  for  each 
condition. 

The  following  gives  the  training  manual  for  participant  in  the  game  controller-discrete  ICS-one 
camera  view  condition. 
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Army  Research  Institute 
Intuitive  Means  of  Robotic  Control 
Testbed 


Participant’s  Guide 

IcamdiscreteGC 


INSTITUTE  for 
SIMULATION 
Q TRAINING 


MAV  Study  -  1ST  1 
Spring/Summer  2006 


(The  format  of  this  manual  has  been  slightly  modified  to  meet  publication 
guidelines) 

Introduction  to  the  Micro-Unmanned  Aerial  Vehicle  Study 
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The  US  Army  is  undergoing  a  major  transformation.  One  element  of  the  transformation 
is  the  introduction  of  a  new  class  of  military  platforms  known  as  unmanned  air  and 
ground  vehicles  (called  UAV’s  and  UGV’s).  A  major  benefit  of  these  unmanned  vehicles 
is  that  they  can  perform  reconnaissance  missions  and  survey  areas  contaminated  with 
radiological,  chemical  or  biological  agents  without  risk  to  human  life.  They  can  also 
survey  the  battlefield  and  provide  real-time  video  feedback. 

We  are  investigating  the  design  of  operator  control  systems  for  micro-unmanned  aerial 
vehicles  that  can  perform  these  kinds  of  reconnaissance  missions.  In  addition,  we  are 
investigating  operator  training  requirements.  In  this  experiment  you  will  be  trained  on 
how  to  fly  a  simulated  micro-UAV  (MAV)  and  then  you  will  complete  a  set  of  missions 
that  will  test  your  ability  to  maneuver  the  MAV  and  locate  various  targets.  After  each 
mission  you  will  be  given  a  short  questionnaire  that  asks  you  to  rate  certain  aspects  of 
the  task  you  performed. 

It  takes  approximately  90-120  minutes  to  complete  the  experiment.  No  previous 
flight  experience  is  necessary  to  participate  in  this  study. 

Confidentiality 

Your  identity  will  be  kept  confidential  to  the  extent  provided  by  law.  Your  information  will 
be  assigned  a  code  number.  The  list  connecting  your  name  to  this  number  will  be  kept 
in  an  electronic  file.  When  the  study  is  completed  and  the  data  have  been  analyzed,  the 
list  will  be  destroyed.  Your  name  will  not  be  used  in  any  report. 

If  you  are  prepared  to  participate  in  this  experiment,  please  read  and  sign  the  Consent 
Form  and  Voluntary  Agreement.  Please  also  indicate  on  that  form  your  preferred 
method  of  compensation.  We  offer  cash  payment  or  course  credit.  Also,  please  feel  free 
to  ask  the  experimenter  any  questions.  Keep  in  mind  that  you  do  have  the  right  to 
withdraw  from  this  experiment  at  any  time,  for  whatever  reason. 

When  you  have  finished  reading  and  signed  the  voluntary  consent  form,  begin  by 
reading  the  following  section  titled  ‘ Overall  Description  of  the  MAV  Simulation’  on  the 
following  page.  You  are  not  required  to  memorize  all  the  information  contained  here,  but 
if  you  have  any  questions  on  the  material  please  feel  free  to  ask  the  experimenter  for 
clarification. 


Overall  Description  of  the  MAV  Simulation 

You  will  be  working  with  a  simulation  for  flying  a  micro  unmanned  aerial  vehicle.  The 
micro-UAV  itself  will  be  referred  to  from  now  on  as  “the  MAV".  This  is  not  a  fixed-wing 
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aircraft  like  most  airplanes  are;  rather  it  is  a  small  rotary  craft  with  an  internal  fan  and 
duct  design  (see  prototype  photos  below).  An  operator  controls  the  MAV  using  a  laptop 
computer  equipped  with  an  input  device  such  as  a  mouse  or  joystick  controller.  This 
interface  is  referred  to  as  the  QCU  —  Operator  Control  Unit.  This  is  a  dismounted  control 
unit,  as  it  is  envisioned  that  a  dismounted  Soldier  (on  foot,  rather  than  in  a  vehicle)  will 
be  controlling  the  MAV. 


MAV  prototypes  from  Honeywell  and  Allied  Aerospace 
Introduction  to  the  QCU  and  MAV  Camera  System 

The  MAV  is  equipped  with  a  dual  camera  system.  When  the  vehicle  flies  through  the 
simulated  environment  you  will  be  able  to  view  video  images  sent  back  to  the  OCU.  You 
will  be  instructed  on  how  to  operate  the  cameras  as  well  as  how  to  use  the  OCU 
interface  and  controllers  to  pilot  the  MAV.  You  will  also  have  an  opportunity  to  practice 
some  manual  flight/piloting  techniques  before  beginning  the  actual  experiment.  After 
basic  instruction,  a  training  session  will  take  place;  then  you  will  move  on  to  the 
assigned  pilot  mission  tasks  where  performance  data  will  be  recorded.  Be  sure  that  you 
understand  the  objective  of  each  mission  before  starting  a  trial.  The  experimenter  is 
available  to  answer  your  questions  before  you  begin  each  task,  so  please  ask  for  help  if 
you  are  unsure  of  any  requirements.  Unless  instructed  otherwise,  it  is  important  that 
you  complete  each  task  as  quickly  and  efficiently  as  possible. 

At  the  end  of  the  training  session  and  at  the  end  of  each  mission  you  will  complete  a 
short  computer-based  questionnaire.  In  the  first  section  you  will  rate  different  aspects  of 
the  task  you  performed;  then  you  will  be  asked  to  choose  between  a  pair  of  items  that 
relate  to  your  performance.  You  must  choose  one  and  only  one  item  for  each  pair.  Then 
you  will  proceed  to  the  next  mission. 


Training  Session 

The  goal  of  training  is  to  familiarize  you  with  the  flight  characteristics  of  the  MAV,  and  to 
give  you  an  opportunity  to  practice  piloting  the  MAV  in  manual  mode.  We  will  begin  by 
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reviewing  the  features  of  the  OCU  and  then  proceed  to  a  series  of  practice  exercises. 
The  experimenter  will  facilitate  this  training  session  and  provide  instruction  on  how  to 
complete  the  assigned  tasks. 

OCU  Layout  and  on-screen  controls 

Below  you  will  see  a  sample  layout  of  the  OCU.  On  the  left  side  of  the  screen  is  the 
video  sensor  imagery  (camera  views),  and  in  the  upper-right  you  will  find  an  overhead 
map  view  of  the  terrain  database.  Just  below  the  map  view  there  is  a  control  pad  that  is 
used  for  issuing  flight  commands  to  the  MAV.  The  control  pad  will  be  examined  in 
greater  detail  throughout  training.  There  is  an  altimeter  along  the  left  border  area  of  the 
OCU  display,  as  well  as  a  task  bar  along  the  top  that  contains  various  icons.  You  will  be 
instructed  on  how  to  use  all  of  the  relevant  gauges  and  icons  during  training. 


OCU  with  single  view  overlapping  cameras  with  discrete  input  control  pad 
The  experimenter  will  handle  tasks  such  as  loading  mission  files  and  scenarios,  so  you 
only  need  to  focus  on  learning  how  to  operate  the  MAV  itself.  Before  beginning  the  flight 
training  exercises,  we  will  learn  about  the  function  of  the  OCU  in  more  detail.  ' 


Understanding  the  Task  Bar  Icons 

The  upper  task  bar  (below)  includes  the  take-off  (and  landing),  mission  mode,  and 
camera  control  buttons.  There  is  also  a  mission  timer  located  on  the  far  right  of  this 
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task  bar.  For  this  study,  you  will  need  to  know  how  to  use  the  take-off  icon  and  the 
camera  control  icons. 


Take-off  &  Landing  icon 


c — ^ 

You  can  take  off  and  land  by  activating  the  task  bar  icon  for  take-off  and 
landing.  This  is  done  by  pressing  button  (10)  on  the  joystick  controller.  Once  the  take  off 
button  is  pressed  the  MAV  will  automatically  climb  to  an  altitude  of  approximately  60 
feet  above  the  ground  level.  At  this  time  you  may  pick  up  the  joystick  and  execute  the 
take-off  command.  You  will  see  the  red  stop  sign  icon  illuminate  when  you  have 
reached  take-off  altitude.  Pressing  button  10  on  the  joystick  will  now  execute  the  land 
command.  You  may  land  the  MAV  now. 

Activating  Camera  Views  and  taking  Snapshot  photos 


These  camera  buttons  allow  the  operator  to  switch  between  the  available  camera 
images.  This  study  uses  a  2  camera  setup.  On  the  OCU,  camera  image  #1  is  the  view 
from  the  MAV’s  forward  camera  and  camera  image  #2  displays  the  view  from  the 
downward  camera. 

One  of  the  unique  features  of  the  OCU  is  the  ability  to  take  snapshots  with  either 
camera.  Before  taking  a  snapshot,  you  will  need  to  activate  the  camera  that  you  want  to 
take  the  picture.  The  active  camera  image  will  always  have  a  blue  border  on  the  top  of 
the  view  frame,  as  well  as  a  ( )  overlay  in  the  center  of  the  image.  The  corresponding 
icon  on  the  task  bar  will  also  illuminate.  To  change  the  active  camera  view  you  must  use 
the  joystick  to  activate  the  group  of  icons  on  the  task  bar  and  select  the  camera  you 
want. 

To  activate  a  camera  view:  This  is  done  by  pressing  and  holding  joystick  button  #2 
while  you  scroll  through  the  available  camera  views  with  the  directional  pad.  The 
experimenter  will  demonstrate  this  feature  now. 

If  not  already  airborne,  try  taking  off  and  switching  the  camera  view.  Snapshots  of 

targets  can  now  be  taken  using  the  (9)  button  on  the  game  controller/joystick. 


Main  Window  Components: 


Altimeter  (See  vertical  bar  on  the  left  side  of  this  page) 
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The  ruler-like  markings  on  the  altimeter  display  the  altitude  of  the  MAV  in  feet 
above  sea  level.  Red  tabs  may  be  visible  on  the  upper  or  lower  regions  of  the 
display  if  the  experimenter  has  chosen  to  activate  the  altitude  alarm  system.  The 
red  areas  simply  mark  the  altitudes  that  will  activate  the  alarm  if  the  MAV  passes 
into  this  “red  zone”. 

<  The  white  triangle  cut-out  (left)  points  to  the  current  altitude  of  the  MAV.  In  this 
case,  the  MAV  is  approximately  82  feet  above  sea  level. 

The  light  brown  column  at  the  lower  end  of  the  bar  marks  the  altitude  of  the 
nearest  surface  below  the  OAV  (this  is  the  current  ground  level). 

Note!~  In  the  current  example,  the  MAV  is  approximately  82  feet  above  sea 
level,  but  the  ground  level  is  approximately  22  feet.  This  means  the  MAV  is  only 
60  feet  above  ground! 


Manual  Input  Control  Pad  (Discrete  Mode): 


This  manual  input  control  pad  lets  you  control  the  position  of  the  MAV  manually.  For 
this  display,  9  buttons  are  used  as  the  interface  to  the  MAV.  The  four  straight-arrow 
icons  represent  forwards,  backwards,  left,  and  right.  The  curved  arrow  icons  in  the 
upper  corners  show  rotation  of  the  MAV  left  and  right.  The  lower  corner  icons  move  the 
MAV  up  and  down.  The  middle  X  icon  stops  the  MAV.  Your  airspeed  is  also  shown  here 
in  knots.  Max  speed  is  6  knots.  You  will  issue  these  commands  to  the  MAV  by  using  the 
joystick.  Joystick  training  is  next. 


Note  on  Input  Controller  Feedback 

Because  you  are  using  the  joystick  to  control  the  MAV,  the  control  display  will  activate 
when  an  input  is  received.  The  display  provides  feedback  in  this  way  to  the  operator 
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that  a  command  has  been  issued  and  is  being  executed.  Arrow  icons  on  the  discrete 
control  pad  will  illuminate  when  that  command  has  been  entered,  i.e.  when  you  push 
forward  on  the  joystick,  the  forward  arrow  will  illuminate.  The  brighter  the  icon  gets,  the 
faster  you  are  traveling. 

Joystick  /  Game  Controller 

The  joystick/game  controller  is  shown  below,  and  by  now  you  have  at  least  performed 
some  basic  tasks  with  this  device.  We  will  now  go  over  how  to  use  the  joystick  for  all  of 
the  tasks  required  during  this  experiment. 


Controlling  MAV  movement  with  the  Joystick: 

The  left  thumb  stick  controls  movement  forward,  backward,  and  sideways  (or  at 
angles),  but  the  MAV  heading  never  changes  when  the  left  thumb  stick  is  used.  Pushing 
up  on  this  thumb  stick  moves  the  MAV  forward.  Pulling  down  on  the  stick  moves  the 
MAV  backwards.  Moving  the  stick  from  side  to  side  moves  the  MAV  right  or  left  without 
altering  heading. 

The  right  thumb  stick  controls  altitude  and  headina/rotation.  Pushing  up  on  the  stick 
increases  the  MAV’s  altitude,  and  pulling  down  decreases  altitude.  Moving  the  right 
thumb  stick  from  side  to  side  rotates  the  MAV  in  place. 

It  is  possible  (and  expected)  to  use  both  sticks  simultaneously  to  rotate,  change  altitude, 
and  move  at  the  same  time.  At  this  time,  you  may  take-off  and  practice  manipulating  the 
thumb  sticks  for  approximately  1  -2  minutes. 

Using  the  Joystick  to  activate  icons  from  the  task  bar: 


The  buttons  on  the  right-hand  side  of  the  joystick  (#1-4)  are  used  to  highlight  groups 
of  icons  on  the  task  bar.  Once  a  group  of  icons  is  highlighted,  the  directional  pad  of  the 
left-hand  side  of  the  controller  is  used  to  select  the  individual  icon  you  wish  to  activate. 


C-8 


To  select  a  camera  view:  hold  down  button  #2  on  the  controller.  While  holding  the 
button,  use  your  left  thumb  to  press  the  directional  pad  to  scroll  through  the  different 
camera  icons.  When  the  camera  view  (marked  I  or  II)  is  highlighted,  release  button  #2 
to  activate  the  window.  Note  that  the  upper  border  of  the  active  camera  view  window  will 
also  turn  blue. 

Heading  Tape 

Note  that  each  camera  view  window  has  a  heading  tape  located  along  the  top  edge  of 
the  frame.  This  number  indicates  the  current  heading  (based  on  360  degrees)  of  the 
vehicle  with  regard  to  that  camera  image.  Because  the  cameras  have  been  locked  in 
place  for  this  experiment,  the  participant  can  assume  that  the  forward  camera  view 
heading  is  the  same  as  the  MAV  heading.  So  if  the  heading  tape  reads  “270”  then  you 
know  the  MAV  is  facing  due  west. 


0-degrees  is  the  same  as  due  North  on  a  regular  compass.  90-degrees  =  East; 
180-degrees  =  South;  270-degrees  =  West. 

Important  Note!!  -  The  downward  camera  view  is  locked  at  90  degrees  from  horizontal 
(this  is  essentially  straight  down).  However,  when  the  MAV  is  in  motion  the  vehicle  tilts 
in  a  similar  manner  to  a  moving  helicopter.  This  tilting  will  cause  the  downward  camera 
to  point  slightly  backwards,  thus  giving  the  operator  a  heading  reading  that  is  opposite 
of  the  forward  camera  view.  I.e.  if  the  forward  camera  heading  is  0-degrees,  then 
the  downward  camera  heading  will  read  180-degrees  only  while  the  MAV  is  in 
motion. 

Practice  Time:  Now  that  you  have  learned  all  the  functions  of  the  OCU  and  the  flight 
controls,  we  will  complete  a  series  of  practice  exercises  beginning  on  the  next  page. 
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Practice  Exercises 


These  exercises  will  give  you  a  chance  to  practice  the  various  tasks  required  to 
complete  the  missions  in  this  study. 

Warm  up 

Start  this  warm  up  session  by  executing  a  take-off  and  briefly  practicing  the  following 
maneuvers.  You  will  notice  that  inertia  comes  into  play  when  trying  to  stop  the  MAV,  so 
you  will  need  to  learn  to  estimate  things  like  stopping  distances  and  rotational  velocity 
carry-over. 

1 )  Move  the  right  thumb  stick  up  and  down  to  make  the  MAV  ascend  and  descend. 

2)  Move  the  right  thumb  stick  side  to  side  to  rotate  the  MAV. 

3)  Move  the  left  thumb  stick  up  and  down  to  make  the  MAV  fly  forward  and 
backwards. 

4)  Move  the  left  thumb  stick  side  to  side  to  move  the  MAV  laterally.  Note  that  the 
heading  only  changes  when  you  rotate  the  MAV  with  the  right  thumb  stick. 

5)  Activate  camera  1  and  then  activate  camera  2.  Now  switch  back  to  camera  1  and 
take  a  snapshot. 

6)  Land  the  MAV. 

Next  you  will  complete  a  series  of  timed  practice  exercises.  The  experimenter  will 
observe  these  exercises  and  determine  if  you  have  met  the  time  requirement  before 
allowing  you  to  proceed  to  the  next  exercise.  All  mission  and  properties  files  needed  for 
these  exercises  will  be  loaded  by  the  experimenter. 

Practice  Exercise  1 


1 )  Press  OK  (button  10)  to  start  the  simulation  and  timer. 

2)  Execute  the  Take-off  command. 

3)  When  the  Red  Stop  icon  illuminates,  execute  the  Land  command. 

4)  This  exercise  must  be  completed  in  30  seconds  (:30)  or  less. 

Practice  Exercise  2 


9)  Press  OK  to  start  the  simulation  and  timer 

10) Execute  the  Take-off  command. 

1 1  )At  or  before  the  completion  of  take-off,  activate  the  view  window  for  Camera  1 
(forward  view). 

12) Take  a  snapshot  with  camera  1. 

1 3) Activate  the  view  window  for  camera  2  (down  view). 

14) Take  a  snapshot  with  camera  2. 

15) Execute  the  Land  command. 

16) This  must  be  completed  in  40  seconds  (:40)  or  less. 
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Practice  Exercise  3  (with  alarm  active) 


1)  The  upper  altitude  alarm  will  be  set  at  150  feet  and  activated. 

2)  Press  Ok  to  start  the  simulation  and  timer. 

3)  Execute  the  Take-off  command. 

4)  Ascend  to  150  feet  and  trigger  the  alarm. 

5)  Immediately  descend  to  50  feet  or  below  without  hitting  the  ground. 

6)  Ascend  back  up  to  100  feet  but  less  than  150  feet. 

7)  Rotate  the  MAV  360-degrees  without  dropping  below  100  feet.  It  is  required  that 
the  heading  tape  shows  the  number  “0”  after  completing  1  rotation  with  the  MAV. 
The  “0”  must  remain  in  the  forward  camera  view  window  before  landing. 

8)  Execute  the  Land  command. 

9)  This  exercise  must  be  completed  in  1  minute  35  seconds  (1 :35)  or  less. 

Practice  Exercise  4:  rapid  command  execution 

For  this  exercise  you  will  follow  a  series  of  oral  commands  issued  by  the  experimenter. 
After  take-off  and  as  soon  as  the  Red  Stop  icon  illuminates,  you  will  immediately  begin 
to  hear  a  series  of  flight  commands.  Commands  will  be  given  as  fast  as  you  can 
correctly  comply.  Once  the  correct  feedback  is  observed  from  the  OCU  the 
experimenter  will  proceed  to  the  next  command. 

Note:  it  is  not  important  that  the  MAV  travels  any  considerable  distance.  The  purpose  of 
this  exercise  is  to  allow  you  to  learn  the  mapping  of  all  buttons  and  icons  and  their 
corresponding  functions.  The  experimenter  is  looking  mainly  for  the  correct  feedback 
from  the  OCU  control  pad  located  in  the  lower  right  of  the  display. 

Rapid  command  execution  -  Part  A 


1 )  Press  OK  to  start  the  simulation  and  timer. 

2)  Execute  Take-off. 

3)  The  first  series  of  commands  after  take-off  will  be:  9  commands. 

4)  This  exercise  must  be  completed  in  1  minute  5  seconds  (1:05)  or  less. 

The  experimenter  will  now  reload  the  properties  file  and  reset  the  timer. 

Rapid  command  execution  -  Part  B 

5)  Press  OK  to  start  the  simulation  and  timer. 

6)  Execute  take-off. 

7)  The  second  series  of  commands  after  take-off  should  be:  14  commands. 

8)  This  exercise  must  be  completed  in  1  minute  25  seconds  (1:25)  or  less. 

The  next  2  exercises  involve  flying  the  MAV  over  longer  distances  and  following  pre¬ 
determined  mission  parameters.  These  will  be  similar  to  the  missions  you  will  complete 
during  the  remainder  of  the  experiment. 
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Practice  Exercise  5 


In  this  exercise  you  will  pilot  the  MAV  around  the  main  roadway  that  forms  an  oval 
inside  the  terrain  database.  Waypoints  will  be  visible  in  the  overhead  map  view  window. 
Waypoints  are  used  to  determine  the  correct  flight  path  of  the  MAV.  The  experimenter 
will  explain  this  to  you  in  more  detail  while  the  MAV  completes  the  mission 
autonomously. 

1)  The  experimenter  will  load  and  run  this  mission  autonomously  and  will  point  out 
the  Landing  Zone  (LZ)  on  the  (H)  building. 

2)  After  the  autonomous  mission  finishes,  the  simulation  will  be  reset. 

3)  You  must  now  manually  pilot  the  MAV  around  the  gray  pathway  while  remaining 
to  the  left  of  the  4  red  poles  and  landing  in  the  correct  LZ. 

4)  When  ready,  press  OK  to  start  the  timer. 

5)  Execute  the  Take-off  command. 

6)  Complete  one  lap  around  the  4  red  poles  and  stay  over  the  gray  path. 

7)  Land  on  the  (H)  building. 

8)  This  exercise  must  be  completed  in  3  minutes  50  seconds  (3:50)  or  less. 


Practice  Exercise  6  -  obstacle  course 

In  this  exercise  you  have  obstacles  to  navigate.  You  will  also  take  2  snapshots  at  the 
end  of  the  run.  Complete  the  mission  by  flying  through  the  series  of  red  and  green  poles 
and  then  return  to  your  start  point  to  take  the  snapshots. 

1 )  The  experimenter  will  load  and  run  this  mission  autonomously  with  waypoints 
visible.  Observe  how  the  MAV  passes  to  the  right  of  all  green  poles  and  to  the 
left  of  all  red  poles. 

2)  At  the  end  of  the  run,  you  will  see  the  C2  vehicle  parked  on  the  sidewalk.  (This  is 
you!  You  control  the  MAV  from  this  position  inside  the  vehicle.) 

3)  You  must  now  complete  the  course  manually  with  a  few  additional  instructions: 
After  you  finish  navigating  around  the  poles  you  will  need  to  take 
snapshots  of  the  C2V  with  both  cameras. 

4)  When  ready,  press  OK  and  then  execute  Take-off. 

5)  Complete  the  obstacle  course. 

6)  Take  snapshot  from  camera  1 

7)  Take  snapshot  from  camera  2. 

8)  Land  -  but  do  NOT  land  on  the  C2  vehicle! 

9)  This  exercise  must  be  completed  in  5  minutes  (5:00)  or  less. 

You  will  now  complete  a  short  computer-based  questionnaire.  The  experimenter 
will  explain  this  and  give  you  instructions  at  this  time.  Then  you  will  begin  a 
series  of  6  missions. 
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Mission  Protocol  for  ARI/IST  MAV  Experiment  1 

There  will  be  6  missions  for  you  to  complete  during  this  portion  of  the  study.  The 
experimenter  will  instruct  you  on  mission  requirements  and  provide  any  documentation 
necessary.  If  you  are  unsure  of  any  of  these  requirements  please  ask  for  clarification. 
Once  you  begin  a  mission,  the  experimenter  will  have  very  limited  interaction  with  you. 
He/she  will  not  be  able  to  answer  questions  on  mission  requirements  once  you  execute 
the  take-off  command,  so  please  ask  beforehand. 

Mission  1 

This  mission  is  a  repeat  of  practice  exercise  #5  where  you  piloted  the  MAV  around  the 
gray  pathway  while  remaining  to  the  left  of  the  4  red  poles. 

1 )  You  will  manually  pilot  the  MAV  around  the  gray  pathway  while  remaining  to  the 
left  of  the  4  red  poles  and  landing  on  the  (H)  building. 

2)  When  ready,  press  OK  to  start  the  timer. 

3)  Execute  the  Take-off  command. 

4)  Complete  one  lap  around  the  4  red  poles  and  stay  over  the  gray  path. 

5)  Land  on  the  (H)  building. 

Complete  the  computer-based  questionnaire  at  this  time  and  then  proceed  to 
Mission  2. 

Mission  2 


This  mission  involves  using  the  MAV  to  do  reconnaissance  work.  You  will  get  a  handout 
titled  “Mission  2  Intel  &  Recon".  Review  this  with  the  experimenter  and  then  complete 
the  required  tasks.  The  experimenter  may  ask  you  for  situational  updates  at  different 
points  during  this  mission. 

1)  Read  the  “Mission  2  Intel  &  Recon”  handout. 

2)  When  ready  the  experimenter  will  load  the  mission  scenarios. 

3)  Press  OK  to  start  the  mission  timer. 

4)  Execute  Take-off. 

5)  Locate  the  church. 

6)  Use  the  MAV  to  observe  the  church  from  all  sides. 

7)  Determine  who  or  what  is  occupying  the  church. 

8)  Pilot  the  MAV  to  the  landing  zone. 

9)  Complete  the  debriefing  with  the  experimenter. 

Mission  3 


This  mission  repeats  practice  exercise  #6  where  you  navigated  a  series  of  red  and 
green  poles.  You  will  also  take  2  snapshots  of  the  C2  vehicle  at  the  end  of  the  run. 
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Complete  the  mission  by  flying  through  the  series  of  red  and  green  poles  and  then 
return  to  your  start  point  to  take  the  snapshots  of  the  C2V. 

1)  You  must  complete  the  obstacle  course  manually. 

2)  After  you  finish  navigating  around  the  poles  you  will  need  to  take  snapshots  of 
the  C2V  with  both  cameras. 

3)  When  ready,  press  OK  and  then  execute  Take-off. 

4)  Complete  the  obstacle  course. 

5)  Take  snapshot  of  the  C2V  with  camera  1 

6)  Take  snapshot  of  the  C2V  with  camera  2. 

7)  Land  -  but  do  NOT  land  on  the  C2  vehicle! 

Complete  the  computer  questionnaire  at  this  time  and  then  proceed  to  Mission  4. 
Mission  4 


This  mission  involves  using  the  MAV  to  do  more  reconnaissance  work.  You  will  get  a 
handout  titled  “Mission  4  Intel  &  Recon".  Review  this  with  the  experimenter  and  then 
complete  the  required  tasks.  This  is  primarily  a  target  identification  mission.  Once  again, 
the  experimenter  may  ask  you  for  situational  updates  during  this  mission. 

1 )  Review  the  “Mission  4  Intel  &  Recon"  handout. 

2)  The  experimenter  will  load  the  mission  files  and  scenario. 

3)  You  will  have  a  limited  time  to  identify  as  many  targets  as  possible  within  the 
terrain  database. 

4)  Positive  ID  can  only  be  achieved  by  taking  snapshots  of  each  entity  with  both  the 
forward  and  down  cameras,  and  each  entity  must  be  centered  in  the  frame  so 
that  the  center  ( )  overlay  is  touching  part  of  the  entity. 

5)  When  ready,  press  OK  to  begin  the  mission  and  start  the  timer. 

6)  Immediately  begin  looking  for  entities  to  identify  via  the  camera. 

7)  The  experimenter  will  tell  you  to  stop  when  time  has  expired. 

Complete  the  computer-based  questionnaire  at  this  time. 

Mission  5:  Repeat  mission  1  -  This  is  your  final  attempt  to  make  the  best  time 
possible.  (Then  complete  the  computer-based  questionnaire.) 

Mission  6:  Repeat  mission  3  -  This  is  your  final  attempt  to  make  the  best  time 
possible.  When  finished  you  may  proceed  to  your  final  debriefing  session. 
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Appendix  D 

Demographics  Questionnaire 


1 .  Participant  Number 

2.  Today’s  Date 

3.  Year  of  Birth 


4.  Gender  o 

Male 

o 

Female 

5.  Have  you  graduated  from  High  School? 

o  Yes  o  No 

6.  Current  Education  Level 

Grade  school  or  less 

o 

High  School  graduate 

o 

Some  college 

o 

College  graduate 

o 

Graduate  degree 

o 

7.  Is  your  vision  correctable  to  20/20? 

o 

Yes 

O 

No 

8.  To  your  knowledge  are  you  color  blind? 

o 

Yes 

O 

No 

9.  How  often  do  you  use  a  computer? 

Daily 

o 

A  few  times  per  week 

o 

A  few  times  per  year 

o 

Not  in  the  past  year 

o 

Never 

o 

1 0.  Estimate  how  many  hours  per  day  you  use  a  computer  on  average. 

0 

o 

1 

o 

2 

o 

3 

o 

4 

o 

5 

o 

6 

o 

7 

o 

8 

o 

More  than  8  hours  per  day 

o 
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1 1 .  How  would  you  rate  your  computer  skills? 

Novice/Beginner 

o 

Intermediate 

o 

Expert 

o 

12.  Which  of  the  following  computer  input  devices  have  you  used  more  than  once? 

□  Keyboard 

□  Standard  mouse 

□  Touch  pad  mouse 

□  Touch  screen 

□  Roller-track  ball 

□  Joystick 

□  Dual  thumb-stick  game  controller 

13.  Select  any  of  the  following  game  consoles  that  you  either  own  or  have  used  on  a  regular 
basis. 

□  Microsoft  XBOX 

□  Sony  Playstation 

□  Sony  Playstation  2 

□  XBOX  360 

□  Sega  Dreamcast 

□  Nintendo  Gamecube 

□  Super  Nintendo 

□  PC  game  system 

1 4.  How  would  you  rate  your  video  game  skills? 

Novice/Beginner 

o 

Intermediate 

o 

Expert 

o 

1 5.  How  many  days  in  the  past  week  have  you  played  video  games? 

0 

o 

1 

o 

2 

o 

3 

0 

4 

0 

5 

o 

6 

0 

7 

o 

1 6.  Estimate  how  many  hours  per  day  you  play  video  games  on  average. 

0 

o 

1 

o 

2 

o 

3 

o 

4 

o 

5 

o 

6 

o 

7 

o 

8 

o 

More  than  8  hours  per  day 

o 

D-2 


1 7.  Mark  the  selection  tha 

best  describes  which  hand 

you  use  for  each  task. 

Always 

left 

Usually  left 

No 

preference 

Usually  right 

Always 

right 

Writing 

o 

o 

o 

o 

o 

Drawing 

o 

o 

0 

o 

0 

Throwing 

o 

o 

o 

o 

0 

Scissors 

o 

o 

o 

o 

0 

Toothbrush 

o 

o 

o 

o 

o 

Knife  (without  fork) 

o 

o 

o 

o 

o 

Spoon 

o 

o 

o 

o 

o 

Broom  (upper  hand) 

o 

o 

o 

o 

o 

Striking  match  (match) 

o 

o 

o 

o 

0 

Opening  box  (lid) 

o 

o 

o 

o 

o 
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MAY  Training  Evaluation  Worksheet  -  ARI/IST1  OCU 


To  ensure  that  you  have  a  basic  grasp  of  the  MAV  pilot  interface  and  the  available  flight 
commands,  please  complete  the  following  exercise. 

Each  of  the  critical  features  of  the  user  interface  are  labeled  above  with  letters  A  -  N.  Every 
letter  must  be  used,  so  choose  the  best  answer.  Enter  the  corresponding  letter  in  the  blank 
following  each  of  the  item  descriptions  below: 

Camera  Selection  Icons  _ 

Vertical  Velocity  Control  _ 

Take-off  &  Land  Icon  _ 

Horizontal  Velocity  Control  _ 

MAV  Location  on  Map  _ 

Ground  Level  Indicator  _ 

Air  Speed  Indicator  _ 


Altimeter 
Mission  Timer 
Rotational  Velocity  Control 
Heading  Tape 
Satellite  MAP  View 
Current  MAV  Altitude 
Camera  2  Image 
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Appendix  F 

MAY  USABILITY  QUESTIONNAIRE  (IMROC-OCU  EXP:  1  Joystick) 


Circle  the  number  that  best  describes  your  reaction  between  the  2  extremes  given. 


1 )  The  system  1  worked  with  was 
wonderful 

1  2  3  4  5 

6 

7  8  9 

terrible 

10 

2)  The  system  I  worked  with  was 
easy 

1  2  3  4  5 

6 

7  8  9 

difficult 

10 

3)  I  found  this  experience 
satisfying 

1  2  3  4  5 

6 

7  8  9 

frustrating 

10 

4)  The  system  I  worked  with  seemed 

Capable  of  doing  the 

exercises 

1  2  3  4  5 

6 

Unable  to  do  the 
exercises 

7  8  9  10 

5)  I  found  this  experience 
stimulating 

12  3  4  5 

6 

7  8  9 

dull 

10 

6)  The  system  I  worked  with  was 
flexible 

1  2  3  4  5 

6 

7  8  9 

rigid 

10 

7)  The  functions  of  the  on-screen  manual 

control  buttons  (on  the  control  pad)  were 

clear 

1  2  3  4  5 

6 

7  8  9 

confusing 

10 

8)  The  functionality  of  the  buttons  for  switching  between 
clear 

1  2  3  4  5  6 

camera  views  was 

7  8  9 

confusing 

10 

9)  Organization  of  information  on  the  video  display  screen  was 
clear 

123456789 

confusing 

10 
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10)  Current  status  (such  as  Taking  off,  Landing,  Grounded)  messages  were 

adequate  inadequate 

123456789  10 

1 1 )  When  controlling  the  MAV  in  flight  using  the  joystick,  the  device  was 

easy  to  use  difficult  to  use 

123456789  10 

1 2)  As  I  progressed  through  the  missions  using  the  joystick,  my  hands  and/or  wrists  became 
fatigued 

Never  Always 

123456789  10 

13)  When  using  the  joystick  to  enter  flight  commands  to  the  MAV,  maintaining  awareness  of 
individual  mission  objectives  was 

easy  difficult 

123456789  10 

14)  When  using  the  LEFT  thumb  stick  (forward,  back,  left  &  right  movements)  to  move  the  air 
vehicle  while  in  manual  control,  the  air  vehicle  reacted  as  I  expected. 

always  never 

123456789  10 

1 5)  Using  the  RIGHT  thumb  stick  (up,  down  &  rotation)  to  move  the  air  vehicle  while  in  manual 
control,  the  air  vehicle  reacted  as  I  expected. 

always  never 

1  2  3  4  5  6  7  8  9  10 

16)  When  using  the  joystick  to  stop  the  motion  of  the  air  vehicle  while  in  manual  control,  the  air 
vehicle  reacted  as  I  expected. 

always  never 

1  2  3  4  5  6  7  8  9  10 

1 7)  When  using  the  joystick  to  switch  between  camera  views  (highlighting  the  camera  icon 
located  on  the  task  bar),  the  display  reacted  as  I  expected. 

always  never 

1  2  3  4  5  6  7  8  9  10 

1 8)  While  using  the  camera  to  take  snapshots  of  targets,  centering  the  target  so  it  was  aligned 
with  the  ( )  overlay  was 

easy  difficult 

1  23456789  10 
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19)  When  piloting  the  MAV  in  manual  control,  determining  the  current  heading  of  the  MAV 
from  the  360-degree  directional  heading  tape  was 

clear  confusing 

1  23456789  10 

20)  It  was  clear  when  the  air  vehicle  had  landed 

always  never 

1  23456789  10 

21)  The  system  provided  adequate  feedback  when  I  issued  a  command  to  the  MAV. 

always  never 

1  23456789  10 


22)  The  OCU  interface  keeps  you  informed  about  what  it  is  happening 
always 

1  2  3  4  5  6  7  8 


never 
9  10 


23)  Learning  to  operate  the  system  was 
easy 

1  2  3  4  5  6  7  8 


difficult 
9  10 


24)  Remembering  names  and  use  of  commands  was 
easy 

1  2  3  4  5  6  7  8 


difficult 
9  10 


25)  Tasks  could  be  performed  in  a  straightforward  manner 
always 

1  2  3  4  5  6  7  8 


never 
9  10 


26)  Training  materials  were 
clear 

1  2  3  4  5  6  7  8 


confusing 
9  10 


27)  After  this  training,  I  am  ready  use  this  system  to  fly  a  real  air  vehicle 

very  confident 

not  at  all 
confident 

12  3  4 

5  6  7 

8 

9  10 

28)  The  system  speed  was 
fast  enough 

too  slow 

12  3  4 

5  6  7 

8 

9  10 
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29)  System  reliability  was 
reliable 

12  3  4 

5 

6  7 

8 

unreliable 
9  10 

30)  Correcting  your  mistakes  was 
easy 

12  3  4 

5 

6  7 

8 

difficult 
9  10 

31)  Both  experienced  and  inexperienced  users’  needs  were  taken  into  consideration 

always 

12  3  4 

5 

6  7 

8 

never 

9  10 

32)  Do  you  have  any  previous  Remote  Control  R/C  experience?  YES  /  NO  (circle  one) 


If  you  answered  yes,  please  briefly  describe: 
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